Disclaimer: this is an automatic aggregator which pulls feeds and comments from many blogs of contributors that have contributed to the Mono project. The contents of these blog entries do not necessarily reflect Xamarin's position.

May 25

Unity at Hack Week 11

How do we maintain the fast pace of innovation and maintain a stable product at the same time? Hack weeks are a part of the answer. The principle is simple: think of a project you want to do, find teammates, hack for a week, present the result. This year’s Unity Hack Week was the biggest ever, more than 80 projects made the finish line on Friday. And their quality was just astonishing!

Unity Hack Week 11 brought 300 Unity employees to Portugal in mid-May with a mission to spend a week working on whatever they wanted. The projects didn’t have to be related to anybody’s day job or any Unity features currently on the roadmap. The opposite is actually ideal: explore beyond your skill set, experiment and collaborate with people you didn’t know before. You can work without the pressure of considering the standard for Unity features.

Kuba Cupisz, a graphics programmer working out of Copenhagen, is one of the few people who have been to all 11 hack weeks, going back the past 6 years. “The first bootcamp was just a way to work together faster and get Unity 3 out of the door. People really enjoyed the format and wanted to use it to do some really creative work and try working in new teams,” he says. The first hack weeks took place in the Unity offices in Copenhagen, but for the past few years, we’ve grown too big to all fit in there. For Hack Week 10, we tried shipping everybody to a big resort in a place with a nicer climate and food. The experiment was successful, so this time the gathering took place in Quinta da Marinha Resort in Cascaís, Portugal.

Despite the summery weather in most of Europe that week, Portugal was actually fairly cool and rainy, keeping us away from the pool and the golf course. Small teams worked all around the resort, restocking periodically on snacks and catching up on the progress of others. Being in one big place together was great for getting to know each other better. Since we have more than 20 offices around the world, some of us met in person for the first time ever.

So what did all those engineers and UX designers make over the week? During the very intense few hours of Friday afternoon presentations, we saw over 80 two minute videos on things like 3D tile mapping, Adam demo video in VR, 2D animation pipelines improvements or a visual debugger. Some people worked on C# Scriptable Importers and some made a location based AR game called Duck Drop.

A lot of the projects were down to earth practical things that we know you need from all the information we get from you through our feedback page or other channels. The documentation team created better feedback forms. The networking team made more examples for the manual. A few teams looked into improving some of the editor functions like the bug reporter or searching for objects.

Some were proofs of concept for a particular approach to a huge possible feature or editor workflows. One team prototyped storyboard state machines, an authoring workflow for building out the states in your game, similar to animation state machines, that can also help visualize analytics. Another group looked into live collaboration in-editor, they didn’t quite get to Google Docs for Unity, but the results were nevertheless very impressive.

MPU_1561

We also saw a prototype of a visual scripting implementation, based on marking C# functions with specific attributes. These define which nodes are available in the visual scripting window and which output and input pins they have. Another project used functionality already provided by the prefab code and the undo system to enable an equivalent of the functionality requested as nested prefabs in the past. The experience we got working on these projects will help us decide what is the right way of solving a particularly problem in Unity.

Other groups made original applications with Unity. A VR Rube Goldberg machine app included in-VR creation and manipulation of assets with Oculus and the touch controllers. The team integrated a 3D procedural generation technology called Sceelix for this project. Another amazing VR prototype was a real-life orthopedic surgery simulator using Vive. Unity for kids is a mobile app where children can easily create small games by combining primitive objects and basic behaviors. We also saw a painting app for iPad Pro made with Unity that also worked as a texture asset editor for Unity.

You can see a few of the projects in this YouTube playlist. Please keep in mind that these aren’t slick trailers of new Unity features. They were made in a few hours by some very tired developers trying their best to present what they built in less than a week. Right now, we have no idea of where some of the ideas may go. They might be utterly unpractical or lead us to amazing performance, functionality and UX wins. Consider this carefully before you get too excited or worried! Remember that we have a public roadmap, which is the best place to look if you want to see what we’re actually working on shipping to you.

It was so inspiring to be so many in the same room at the same time, watching the results of so many clever experiments. At Unity, you’re never really the smartest person in the room. Instead, we’re all learning from each other.

Do you want to hack with us next time? Check out unity3d.com/jobs!

Quinta da Marinha Resort in Cascaís, Portugal Hackers be hacking First day planning MPU_1856 MPU_1859 MPU_1891 Unity Docs Team Na'Tosha and Levi Bard working on using SDL in Linux runtime. VR Rube Goldberg machine app team Working on animation improvements MPU_2179 Beautiful coast of Portugal MPU_2492 2016-05-11 18.05.25-1 MPU_2620 MPU_1561 Portuguese Paella Ladies night at Hack Week Nobody used the pool like the ducks did Gear VR by the pool MPU_2886 Victory! Storyboards team Selfie tennis Holographic Unity Chan Unity drawing app Final night party

May 24

Breakout! How to stop an infinite loop in a Unity C# script.

(or what I did at my first Hack Week with Unity)

This post is about a little trick for breaking infinite loops in scripts in Unity. It works with Windows / 64 bit in the Editor and in 64 bit builds with script debugging enabled. With a little more effort it can be made to work on 32 bit and even builds with script debugging disabled.

Infinite loops seems to be something that should be easily avoidable. But from time to time, I’ve encountered them in sneaky variants. Once it was the broken random function returning 1.000001 on impossibly rare cases. Another time a degenerate mesh sent a NaN right into an unsuspecting while(1) { d += 1.0; if(d>10.0) break; /* .. */ } loop. And then there was the broken data structure traversed by an algorithm that assumed current = current.next; would surely lead to an end eventually.

If you have experienced an infinite loop in your script code in Unity, you know it is quite unpleasant. Unity becomes unresponsive and you may have to kill the Editor entirely to get out of the mess. If you were lucky enough to have the debugger attached before running your game, you may be able to break it. But usually only if you can guess the right place in the code to set a breakpoint.

Some years back, before joining Unity, I found a way to break a script that is stuck like this. But it was not until my first Hack Week here, that I got to talk to the right people and realized why the trick works and how it may be used to create a proper way to break scripts in Unity (sneak peek of my Hack Week project). Until we get that feature properly done and release it, you can use the trick below. Or just hang on for the fun of a little tour into disassembled, jit’ed code. What could possibly go wrong?

Do not try this at work!

As a trained professional you know the value of practice, so try this out on a toy project before you attempt a rescue operation at work. Fire up Unity and create an empty project, add a box to an empty scene and create a new C# script “Quicksand” attached to the box. The script should contain this code:

using UnityEngine;
class Quicksand : public Monobehaviour
{
            public OnMouseDown()
            {
                    while(true)
                    {
                   // "Mind you, you'll keep sinking forever!!", -- My mom
                    }
            }
}

Now hit play and click the box. Observe Unity freeze up and experience the onsetting rush of panic until you remember that this is just a test. No actual work is going to be harmed!

So now your script is stuck and Unity seems to be hung. Let us start up a new instance of Visual Studio.

For this to work you probably (I didn’t check to be honest) need to have selected C++ as one of the languages when you installed VS. Go to Debug menu and select Attach to Process (NOTE: this is not the same option you usually choose for attaching to Unity). Locate the Unity process and attach to it.

img_02

Having attached the debugger to the stuck Unity, select “Debug | Break all” and find the disassembly view showing the code currently executing on the main thread. The following gif shows the little dance you have to perform. Perhaps you even need to click on “show disassembly” or something like that depending on your configuration of Visual Studio. (On one machine where I tried it, I had to hit F10, which does a single-step, for the disassembly to show).

img_03

As you maybe know, the scripts are — for performance reasons — translated into machine code on the fly before being executed. This is known as jit-compiling (just-in-time compiling). The result is what you see in the disassembly window. Hopefully it looks something like this:

img_04

In this case you can almost see the infinite loop (indicated by the red arrow I have artistically rendered on top of the screen snip). There is one mov, one cmp and a lot of nop’s and then the jmp loops right back to where we started. No escape!

In a more realistic case, your C# code will be more complicated, and it will be harder to tell what is going on, but you don’t really have to understand it, because here comes the trick: hit F10 (single step) a number of times until you get to one of the “cmp dword ptr [r11], 0″ instructions. They should be sprinkled liberally all over the code because they are part of the debugging infrastructure. After a few steps, you may end up with something like this:

img_05

With a bit of luck you already have a window named “Autos” (if not, use Debug | Windows | Auto to find it). It should show you the value of the registers that are in play at this point:

img_06

Now simply change the value of R11 to 0. Like this:

img_07

If you were to execute the cmp instruction now, it would try to read from address 0 which is going to generate an exception. And that is actually exactly what we want: So hit F5 (continue program execution) and answer “Continue” to the dialog box that pops up:

img_08

If all went well, you now get a proper (Mono-)exception in Unity, the loop is broken and Unity is back to normal. You can save your work and look at the call stack in the Console to see which part of your script code caused the hang.

img_09

That’s it. Go forth and loop! A little warning is in place: you have now messed around pretty deep inside Unity and it is prudent to save your work (if needed) and restart the Editor. My experience is that everything seem quite healthy, but just to be on the safe side.

Why does this hack work?

The reason this works at all is that Mono has a built in system for debugging scripts. It works by sprinkling (actually, once for each C# line) the jit-code with reads from a specific memory address. That is the “cmp dword ptr [r11], 0” instructions we saw above. When you are in debugging mode and are single stepping through your code, the page that holds this memory address is made read-only and we will hit an exception once for every C# line of code. The Mono framework can catch this outside of the jit-ed code and basically pause the execution after every line.

The trick we did above, where we set the register r11 to be 0, will end up generating the same type of exception, because the address 0 is never readable. So the debugging framework sees something that looks like single-stepping, but because we are not really debugging, a NullReferenceException is generated and we get a nice call stack. Very convenient!

This technique also works for standalone games. You attach to yourgame.exe, break all, find the jit code, force a memory fault and you should be good. You will have to look up the call stack in the log file, though.

Corner cases

The example we just looked at was, to put it mildly, conveniently simplified for tutorial purposes! In reality, there are a number of catches you may run into. When you hit “break all” you may not actually break into ‘clean’ jit’ed code. If your C# code makes use of any API calls, the program might be inside some of the core Unity code. It will look like this:

img_10

Here we see the program having called into GetPosition. When the top of the Call stack contains real function names and not just hex addresses, it is usually a sign we have left mono / jit’ed code. But just hit Shift-F11 (Step Out) a few times until you are back in jit-land (nop’s galore is also a good indicator of jit-code).

Sometimes you can manage to break Unity at a point where the main thread is not active. It is probably easiest just to continue (F5) and then break all again until the main thread is active.

There are probably more weird cases, but hey, this is debugging, so improvisation is key!

What about 32bit

You can do something similar in 32bit mode. The jit-code looks a little different. Perhaps something like this:

img_11

This means we read from 0xB10000 in this case. To provoke a page fault, you need to actually change the code because the address is hardcoded directly in the instruction and not in a register as with 64 bit. So you open a memory view (Debug | Windows | Memory | Memory1) and navigate to the address of the instruction (the yellow arrow), that is, 0x65163DC in our case. Here we find:

img_12

You can recognize the address: it is the “b1” found 4 bytes in from the start. Change it to 00 and then continue (F5) as before. This will have the same effect, but with the difference from the 64 bit case, that you will break out every time you hit this location.

So what about non debug mode?

Ok if you are really unlucky, you may have a bug that is only reproducible when you compile your scripts with debugging disabled. In this case you have to improvise a bit. If you can look at the code and find a way to provoke a read fault you should be golden, but it may or may not be super easy. As a last resort you may need to inject, manually, something like cmp eax, dword ptr ds:[0x0] which we know from above happens to be the sequence 3b 05 00 00 00 00. But maybe we are a bit more lucky. Let us try our example script from above. Breaking it yields:

img_13

Oh no! The worst. The compiler optimized it to just one jmp instruction looping on itself. There isn’t even room for adding in our cmp (the jmp is relative and only takes up 2 bytes). However, since we are assuming everyone is desperate (a release build hung, after all) we don’t have to be too careful and can just trash the code with our read. Navigate to 4D34446 in the memory window and fill in 3b 05 00 00 00 00 on top on whatever is there. Hit continue (F5) and hope. In my case the game (I was doing this part of the test with a standalone game)  came back to life and I could inspect the output log to find:

img_14

At this point you really should shut down the game as you have effectively ruined a part of the jit generated code and your scripts will likely not work anymore. But at least you know where you were stuck.

Sometimes you can find a read instruction in the vicinity of where you stopped. Then you can right click on it, select “Set next statement” and by setting a register to 0 you may be able to create the right exception that way.

Conclusion

So with a bit of trickery it seems unbreakable loops are in fact breakable. Hurry up and try it out so you can join the ranks of grizzled veterans growling “Ha! In my days we did it in disassembly!”. Soon we’ll ship a better solution and it will be forever too late!

May 19

Make your Daydreams a reality

It was really exciting to see John Riccitiello, our CEO, on stage today at Google i/o with Clay Bavor as they walked through the details about their new Daydream VR product.  We believe that Google shares our vision of democratizing development, and ultimately democratizing VR, which is what makes our partnership so exciting. 

Furthermore, we think mobile is going to be a key part of that, which is an easy conclusion to reach just by doing the math. With an installed base of around 3 Billion in 2016, mobile devices are already dwarfing the 1.5 Billion PCs.  That makes Daydream particularly interesting from a mass market perspective and it very smartly fits into our strategy of creating a platform that allows any developer to create once and publish everywhere.

You may have heard us talk about the “gap of disappointment,” or the slower than suspected start to VR adoption/content this year. Daydream, as well as other platforms like Samsung’s GearVR are going to help drive what we ultimately think will be massive adoption of VR globally on the scale of Billions of people 10 years out. That’s no small number, and we believe it will be just that big.

We also announced today that we will be supporting Vulkan, and plan to give early access to developers in Q3 of this year.  We’re committed to solving hard problems for our developers, and we believe these announcements are aimed at doing just that.  In the end, we want people to create amazing experiences that will delight consumers across the globe.  It’s an amazing journey we’re on, go out and dream, then use Unity to bring those dreams to life.

How Next Games unlocked the secret to ad success

‘You can’t put ads in a battle builder,’ Next Games’ peers told them. ‘No one’s ever done it and it will never work.’ This was the general message that Next Games – one of the stars of Helsinki’s thriving mobile games scene – got from their fellow developers when they pegged them for feedback on their monetization plans.

Rather than abandoning the idea of using a view-to-play model in their two breakaway hit games – the IP-based battle builder, Compass Point: West, and the strategy game, The Walking Dead: No Man’s Land, based on the popular comic book and AMC TV series – Next Games decided to go about it in a different way. Instead, Next Games put extra effort into incorporating the video ads in a way that added to the overall game experience, enticing players to immerse themselves even deeper in the game.

West_logo_color

As Saara Bergström, Next Games’ CMO, put it, “the increased engagement and retention from a good video ads experience puts money in our pocket from the increased lifetime value of our players.”

After some monetization research, Next Games knew that video ads, if integrated in the right way, would indeed work in both their titles. The trick was that they had to design and fit ads into the natural gameplay experience.

Next Games implemented this approach by integrating rewarded video ads into their games, using their own lead game designer to make them feel like a natural part of gameplay. What’s more, the technical integration was – as Bergström said – “remarkably painless.”

Find out how Next Game’s players have embraced rewarded video ads and both engagement and retention has skyrocketed.

Read the full case story

Screen Shot 2016-05-19 at 14.13.24

May 18

The Unity Engine QA Process

We often get requests to explain how we test the engine before we release it. We have previously blogged about it, but none of these posts give a clear overview of the entire process. It’s a large undertaking to do in just one post and for that reason this will be a series of blog posts.

This first post is going to outline the structure of our test efforts and subsequent blog posts will be made to describe details of each individual part, always linking back to this overview. It is also worth noting that every strategy comes from the product and deployment model, so our process on the engine is very different from the process we use on our services. This blog series relates only to the Unity engine itself.

In the simplest form, our releases go through the following lifecycle:

image04

This is what is most obvious to our users, because our version numbering follows the same structure. With this frame we can start exploring what happens at every stage of the product from a QA point of view.

image07

Before we even get code into trunk where we integrate all the features, we do testing on the feature in a branch based off of our mainline source repo, trunk. This is typically feedback on the usability of the feature and ensuring it doesn’t introduce massive instability. How this is done effectively is a topic for a blog post of its own.

image03

Once the feature has been tested in-branch, the developers create the initial Pull Request (PR) in which a code review is done on the code, all automation has been executed and proven to pass, a review of the test cases written is done and the Release Manager does a risk assessment before merging it to trunk. A lot of this is automated through our merge queue process, but it can involve 20-30 people to get a big feature into trunk. And remember, this process is done on every single PR going to trunk, not just one time.. All of these are blog topics for later.

image00

In the alpha phase, it is all about integration of all the features done in the branches. Before we send the first alpha to the alpha list, we run our Release Acceptance Test (RAT), which is a smoke test covering the surface of the platforms and the editor to make sure we have human eyes on as many parts as possible before we ship.

During the following alphas we get more branches trickling in and we continue our integration testing on the builds. During this time we also get a lot of user feedback on the usability and viability of the features from our alpha users. This sometimes results in a feature being cut completely from the release.

In the end of the alpha phase we have an Exploratory Test (ET) week. This approach was  covered by Claus in 2013 and the format remains the same today.

image05

The beta phase is now fully public, so we ship the builds directly to all users. Creating the engine with our community has been a key part of how we developed Unity through all the years and it is still an integral part of every release.

During the beta phase we do a Full Test Pass (FTP), which is a manual regression test pass of all our manual test cases, which test the various workflows and runtime features, spread across all the supported platforms, so we get as broad a surface as possible with human eyes and brains on all of it.

The FTPv2 is a special session we carry out to combine the workflows to make actual games with the build. It is a fun and intense way of testing Unity.

The Asset Store Bash is three days where we go through a list of Asset Store packages compiled from the top 500, upgrade the projects and see if they reveal problems in the builds.

The documentation bash is a few days where we go through our documentation in order to find bugs and discrepancies between the docs and the actual functionality in the engine.

We also execute upgrade tests, in which we take large user projects and upgrade them. It is surprisingly difficult to perform this test and we most certainly need a blog post explaining why this is so. We realize it is a source of a lot of user pain, so we invest a lot of effort in this.

image06 (1)

The release candidate phase is started off by an RC Test Phase where we do a risk driven pass on the engine. This risk assessment is done by the dev teams involved in each area and we use this to drive a targeted effort to get more comfortable with the state of the product before release.

We also run RATs on every one of the RC releases, because they are candidates to be released. As such we treat all of them as the last one.

Screenshot 2016-05-17 13.54.10

In the lifetime of a release we are constantly getting bugs reported by our users. Our community and users are the most important asset of Unity and we value this contribution more than anything else in the process of making a Unity version come to life. It is definitely not an easy task and we have posted multiple blog posts about how we handle this flow of bugs. Best place to read about it is The Great Incident Bash of 2015.

Throughout all our releases in the entire lifetime of the release we verify all bugs fixed. During verification we go through the steps to reproduce the bug, look around the immediate area around the fix to find fallout and then close it or reactivate it depending on the result.

During the lifetime of the engine, we also run our performance regression test suite multiple times. Sakari made a two part blog post about it two years ago and lots has happened since. It is a tough problem to solve consistently and we have invested a lot of effort into this, so we will make a part three to update you on what is going on these days.

It’s important to also understand that testing a platform is a separate discipline in itself. We are helped by having lots of our automation be able to run on all our platforms, but there is a lot of custom manual testing needed for each platform because they all have their independent functionality to consider.

image01

Once a version has been shipped, it goes into sustained engineering (SE) mode. This team was founded two years ago and a lot has happened since. They are responsible for all the patches we send out and we will most certainly create an updated blogpost about how this team is now functioning.

I hope this sheds some light on the overall process of shipping a piece of software as complicated as Unity. We will post a lot more blogs about the details of all these parts. The automation we are doing is such a big part of our process and ability to ship all these platforms, so we will create a similar overview post dedicated to this.

May 17

Unity Beta Tips: How to participate effectively in the beta program

As part of our ongoing commitment to both stability and innovation, we are unveiling some new and improved resources for Unity beta participants. These resources will help new and existing beta testers get the most out of the program, and ensure that the next stable version of Unity works for your project.

We’ve updated the beta page to include more information about how to participate in the beta program. In a nutshell here’s what we’ve got for you:

If you missed what’s coming to Unity 5.4, head over to this blog post and check out the new features and updates you can look forward to in the current beta.

We’re very excited to hear what you think about the new features, but it’s also important that we ensure existing projects continue to work well. This is why we are encouraging you to try Unity 5.4 on a copy of your projects first.

While our engineers test each beta version with numerous projects weekly, it’s only a fraction of the projects out there that are constantly pushing the engine in new and different ways. The best way to make sure the next stable version of Unity works for you and your project is to try it out with the current beta.

If you encounter a bug, be sure to let us know. To make sure you have the right information on how to submit a solid bug report we have created a “Guide to Being an Effective Beta Tester”. It takes you through the process of using the beta and reporting a bug, start-to-finish. The guide covers:

  1. Installation and getting started
  2. The beta forum
  3. Documenting and reporting your bug
  4. Following up

With that, you will be equipped to provide actionable feedback, so we can address bugs more quickly and effectively.

We also have a quick survey to better understand why you do (or don’t) participate in the beta program. Your opinion matters. All of that information is available on the beta page, along, of course, with the beta download link

Monologue

Monologue is a window into the world, work, and lives of the community members and developers that make up the Mono Project, which is a free cross-platform development environment used primarily on Linux.

If you would rather follow Monologue using a newsreader, we provide the following feed:

RSS 2.0 Feed

Monologue is powered by Mono and the Monologue software.

Bloggers