Disclaimer: this is an automatic aggregator which pulls feeds and comments from many blogs of contributors that have contributed to the Mono project. The contents of these blog entries do not necessarily reflect Xamarin's position.

May 3

Analytics & The Player Lifecycle

Using data to improve your game’s retention and revenue

The innovations which have elevated the games industry over the last year haven’t just been about the free to play model. In fact, arguably the most important change has been the rise of games as a service and in particular the use of Data.

I’m lucky that for much of my career I was the games guy in Telecom companies so I was exposed to the love of data very early; but for many this topic is still new. In this post I plan to give you a quick starters guide to practical ways to use data to reduce player Churn and improve both revenues from Ads and in app purchases. More than that though I want to show how important it is to use data to create a dialogue with your players and allow you to respond to their changes as they become more engaged with your game.

First thing to understand is that players’ reactions and choices aren’t static; they fluctuate over time. There are patterns but these are specific to each game. If we want to understand how to motivate players effectively, we need to understand where in that journey they are and how different groups of people react at those lifecycle stages. I break these down into four sections, Discovery, Learning, Engaging*. Understanding what (and why) players actually do at these stages is key to determining our best response. In this session I will also go into more detail on the Engaging stage itself and look at the specific of Super-Engagement and Re-Engagement as well. A video of the webinar recorded on Tuesday, May 3rd 2016 can be found at the end of this blog.

Terms of Reference

Before we dive into the player lifecycle, we need to make sure we have a common set of terminology and an understanding of the kinds of data we can access and what they mean.

Type of Data What is is? How we can use it?
Geodemographic Age/Gender/Location/Household type Knowing who our players are
Behavioural Events recorded within the game Knowing what players did
Operational Performance data of our systems Knowing how stable the game is
Cohort Data A group of players with similar set-up Creating a common base-line
Funnel Analysis A way to monitor stages of play To locate where we can improve
Qualitative Personal feedback from players Understanding Rationale of Behaviour

How Do We Capture Events?

In order to get the behavioral data we need, we capture the events or activity the player actually did in our game. Unlike qualitative research, we aren’t looking for opinions or vague memories of players; we need to know what specific actions/choices they actually made. To do that, we create a series of events which are typically player initiated trigger points which tell the game to capture some values.

For example, we might wish to capture that a player was shot in a FPS game. We might give that a name, say PlayerShot, and then store a range of variables with that event. This could include the Date and Time that the hit was resolved, an anonymized PlayerID, the X,Y,Z coordinates of their Avatar, the damage taken, and the anonymized Player ID of who fired the shot as well as the Session ID for that game.

Importantly we don’t have to capture everything. There are some kinds of data which are static reference information.  For example, the specific position in a specific map. As long as the version of the map used at that time is known, then X,Y,Z coordinates alone can be used to create a heatmap later. We can also infer a lot of data from other events as long as there is some connecting information. For example, we don’t need to capture the Level that the player is using for that game in every event or even a list of all the players in that session.  We can capture that information with a specific Start Session events and use the associated Session ID to allow us to identify everything that happened in that specific game session.

Notice we want the players to remain anonymous. We don’t want or need to spy on our players, but we do need to understand how the game plays across all players.

What Events Should We Capture?

Because we are trying to understand the Lifecycle of our players it’s important to think of the game events in terms of the timeline in which I might encounter them. I find it useful to map out the player experience from the flow of the first time user and separately from the flow of the repeat user. Personally, I’m not necessarily looking for every button press, I’m looking for moments which include meaningful choices. There is an approach used by the Food industry called HACCP. I wrote about this for Develop Online, so I won’t go into detail here, but in essence the point is that we are looking for the “Hazards” such as whether they churn (i.e. leave the game) but also trigger points for more positive action such as paying for an IAP or watching a Video Ad.

It’s also important to recognised that the data we collect will be incomplete, for example if the battery dies or the player switches to a phone call – we will probably not get the last upload. This is less of a problem with a server-based game but it’s never 100% and the compromise is that the game can’t be played offline; impacting our chance for them to create habits of play.

Some typical events could include the following (apologies for the use of pseudocode variable names):

Event Data Points Captured
GameMenuLaunch: AnonPlayerID; TimeIconLaunched
SessionLaunch: TimeSessionLaunched; AnonPlayerID(s); SessionID; LevelIDSelected; OptionSelected
SessionStart: TimeSessionStarted; AnonPlayerID; SessionID;
ObjectiveSet TimeObjectiveSet; AnonPlayerID; SessionID; ObjectiveID;
ObjectiveMet: TimeObjectiveMet; AnonPlayerID; SessionID; ObjectiveID; Score; Reward;
XYZLocation
TargetHit: TimeTargetHit; AttackerID(AnonPlayerID?); SessionID; TargetID(AnonPlayerID?);

Damage, XYZLocation

PlayerDeath: TimePlayerDeath; AnonPlayerID; SessionID; XYZLocation
LevelComplete: AnonPlayerID; SessionID; ObjectiveID; Score; Reward; XYZLocation

From creating events in this way we can infer a huge amount of information. For example, if we want to know the percentage of players who complete a level we can count the number of GameMenuLaunch events with the number of LevelComplete. But we can also get smarter with our analysis. We can look at how many people completed a specific ObjectiveID in a specific LevelIDSelected and compare that to the number of LevelComplete in the subsequent level to find out if skipping objectives in earlier levels have a particular impact on performance later.

It’s not just about comparing events in terms of number, but by looking at the timestamps of specific activities in our game we can identify other deeper issues. For example, if we start to notice that the amount of time spent on the menu page starts increasing, this might be warning sign that they are about to Churn or that changes to your menu page are confusing players.

What Don’t We Know?

Data capture like this can be very powerful, however, the way you capture information will be inherently biased towards your understanding of the way the game is meant to be played. This means that we have to constantly review our metrics. The biggest issue is that we can’t capture what players will do or might have done. That might seem obvious, but we will be using insights from analytics to inform our design decisions. If it’s true that only 2% of players spend in a free to play game, then that means we don’t know what would have triggered the other 98% to spend. There might not be anything that would have convinced them, but the point I’m making is that the data we capture cannot ever be complete and that means we haves to consider statistical significance and be very wary of assuming causation rather than correlation.  Looking at the lifecycle of a player helps us mitigate this, because we can break down each decision stage and look for ways to increase the likelihood of converting players at each stage. It’s all about asking the right questions at the right time.

Thinking Funnels

There are some great techniques out there to help us look at data as part of the player lifecycle. In particular one of the most useful is called Funnel Analysis.  This method allows us to look at a set of players and identify how many of them go through to the next stage of the game. The most often quoted version is the ARM funnel.

With this we look at the number of players we acquire (often as total downloads, although the number of GameMenuLaunch is arguably more useful) through organic installs, direct advertising or virality. Then we compare that with the number of those players who also play the game on subsequent days.  When we compare players on the second day (which I call D2 – others call D1) with players on the 7th day (i.e. D7) or the 30th day(d30) we can get a pretty good idea of what our retention looks like.  Then comparing that with the % of players who pay, we get a good idea of our success in terms of monetization.

However, this doesn’t give me enough detail. So when running a game as a service, it can be useful to expand this funnel to reflect the player’s lifecycle – I call this the Service Funnel. This helps because it allows us to think about the role of the Engaged player as helping feed not only virality (which obviously has diminished over the last few years) but also the willingness of others to spend more in our game. This is why Freeloader players actually add hidden value in the game – as well as their income from, for example watching rewarded Video Ads.

This way we can better map the flow of the gameplay with the way that we engage and retain users; and more importantly look at the factors which create a Repeat purchase in the game.

service funnel new

Can We Make Better Games?

All this may seem quite commercial and dry, indeed there will be some designers who will think that this kind of approach kills creativity. Actually, the reverse should be true. This is about using evidence to create the most engaging game. We want our player to continue playing over time and if they feel good about spending in the game then they will want to spend money more than once (as long as we continue to give them value in doing so).

The breakdown of our game into stages of engagement matters and allows us to use gameplay, Ads and In-App Purchases to increase enjoyment of play.

However, data has to inform the designer, not become a barrier – qualitative research can help. Asking players and observing them using your game can be invaluable as long as you understand the findings are usually only useful to help you understand motives. Focus groups and online surveys help provide some insight but remember that players are essentially incapable of accurately telling you about what they did, let alone what they will do; but they can tell you how they feel about your game and why they made certain choices. However, this needs to be done in a formal setting  – asking your friends down the pub isn’t usually that useful. Direct feedback can be very useful but again we have to be careful about how we use it as it will usually describe the game that the specific player would have made, not what the wider audience need. However, it’s always important to listen and understand.

Ads retention

What Are We Looking For?

Every game will be different, the following is what I’m generally looking for in terms of the performance of the game and the behaviour of players at each of the player lifestages for each. The key question is about how we get players into that stage and what we need to achieve to transition them to the next stage.

Discovery

Discovery is the very initial transition stage from players becoming aware of the game to their action to install it on their devices. At this stage, it’s key for us to understand if we converting Players to Download; but also to actually play the game!

  • Attribution – we need to know not just which sources give us users; but which bring value for retention and monetisation as well. Where you can track these as separate custom segments and compare behaviour.
  • Always compare players by their elapsed time playing.  Often bringing Custom segments with the same start date works really well; although it can also be useful to compare players based in the number of days of play (even if not concurrent)
  • Whilst many ad networks use a combination or CPI or eCPM for calculating the cost of advertising campaigns, we need to look at the effectiveness of every campaign in terms of its % Conversion To Download
  • Downloads aren’t the end of the line – do you know what % Ever Launch the Game or how long it takes them from Launch of the App To playing the game?

Learning

Learning is the first time user experience for the player, but only ends when playing this game becomes routine. That’s vital, as we are looking to truly engage the player and help them become fans. The core question we must ask is if players understand how to play and how the game fits into their lifestyle!

  • What is the retention on the second day of play (I call this Day 2 Retention)? If we can’t get them back the next day, the game isn’t going to be interesting enough to them to pay for content. Even if this is a premium game abandoning the game so early is a bad sign.
  • What is the Frequency & Duration of Sessions? This is going to vary with each game and there is no one level which is ‘good’ or ‘bad’. However, what we are going to be interested in is how this rate changes over time.
  • Playing the Tutorial is rarely fun.  But do we know what % Initiate Level 2? Or later levels. The rate and pace of unlocking new content can indicate the genuine interest in the game.
  • During the Learning Stage I’m not all that keen on converting users to paying – largely the early payers are often one-time payers.  However, it is essential that we set an expectation of value for the goods.  One way to check that is to understand the % Who Watch Video Rewarded Ads
  • It’s often useful to keep track of the Last Position Played in a game session. This will often indicate natural breaks and places where you should have a ‘Call Back Action’ i.e. a reason to set an appointment to return the game at a later date.

Engaging

At the Engaging Stage we really have a true fan. This is the point where we really need to focus on retention and expectation of value. As a designer, I value “Repeat Players and Payers”.  I see one-time payers as a red flag that something is wrong with my monetization. Key to revenue generation is to create an expectation of value in the player. Show them the value of the items (often assisted by access to premium elements through rewarded video Ads). However, the key question we need to ask is: are they ready to pay?

  • The key metric is still retention. The impact of long term commitment on willingness to pay and how much is linked to how long they keep playing.  This may sound cynical but it really isn’t. It’s a reminder that we need to make a game so good, so enjoyable that player will want to keep playing. What is your Day 7 Retention?
  • Every game is different so there are no hard and fast rules about % Conversion To Pay. However, it’s important that spending money in a game should feel great; and not victimise non-payers. Imaging playing Scissor-Paper-Stone. If I buy the Lizard-Spock upgrade this gives me more choices and all other players an exciting twist. But we can all still play together.
  • Just as during the learning phase, the frequency of Watching Video Ads can give great insight to the attractiveness of your IAP and if it rapidly changes, this can help identify a willingness to make the initial conversion to buy (or indeed a risk of churn). It’s likely that your biggest spenders will also be heavy ad users too.
  • Pay attention to the average level of success and failure of your players during the Engagement phase. Frustration can be a great driver to commitment, but it can quickly turn against you.
  • Pay attention to any obvious deviation from Average Success/Failure Ratio.
  • Another neat trick is to map the first point in the game where a player decided to watch a Rewarded Ad or paid for IAP. It can be really useful to understand why – not just where. We need the experience to be fabulous if we want them to do it again!

adsengagement

Super Engaging

One of the things that separates a game with good IAP from a premium game is the ability to people who really fall in love with the game to want to invest more into the experience.  Again, this should not be a cynical exercise but we should be able to work out through analytics not just which Players are ready to get serious, but more importantly what is it they really value.

  • Again, my personal choice is to look at Day 15 Retention; this is a very serious level of commitment regardless of whether they are spending or not.
  • Keep an eye on the % of Repeat IAP Purchase as well as the frequency & changes in Types of Purchase. This is important as especially of you can identify better value goods which help extend retention even further. This about why the goods are likely to give players a reason to return. It’s important that players feel they really got value for what they spent money on.
  • Also look at the level of social communication. Social capital is an essential element to the perceived value of items in your game, just as much as their gameplay value.

Re-Engagement

Players will stumble and fall out of the game, but we should look for ways to stimulate their enjoyment and ensure that they continue to get the most from the experience. If we can pre-empt a problem or frustration point, perhaps even offer new content this can help players keep doing what they love about our game. But sometimes they may just be ready to churn.

  • Players are normally stimulated by an update, but look at the different cohorts of players to see if some fail to re-engage. Any change in behaviour after updates can provide important insight.
  • Watch out for the Rate of Change in frequency of sessions, purchase/Video Ad behaviour, Social Communication, even the Average Success/Failure Ratio. If this slows, can you act to stimulate interest?

Churning

Players leave.  It’s inevitable. Our job is to keep their interest as long as possible and give them a stimulating, entertaining playing experience as long as possible. Recognizing that the time has come is as important as recognising when we can re-engage players

  • What was the pattern in terms of rate of change of session length, time between sessions, response levels to updates prior to the last play session
  • Are those players still willing to recommend the game to others
  • How would those players rate the game now they no longer play? This is key for me. Because the more players who leave and remain happy, the more better chance they will return for the next game.

April 28

Games by the Numbers (2016 Q1): New mobile gaming report reveals current trends

Unity Analytics released the second edition of the “Games by the Numbers” report, which employs data to reveal game install trends across different platforms, devices and regions.

Key highlights from this report:

  • Strong first quarter: Mobile gaming rang in the new year with a 30% increase in Q1 2016 compared with Q4 2015. One country in particular, China, represented 31% of the total installs of games made with Unity.
  • Android still ahead:  Android retained its status as the global leader in Q1 2016 representing 81% of mobile game installs while iOS maintained 17%.
  • Leading Android platforms: Kitkat (4.4) and Lollipop (5.0 and 5.1) account for 73% of all Android game installs.
  • iOS shows more consolidation: 72% of installs are on devices running iOS 9.0 (released September 2015) or later, and 90% on devices running 8.0 (released at September 2014) or later.

Blog-body-image-800x200 (1)

To see all the insights and trends, download the full report:

DOWNLOAD FREE REPORT

Since the beginning of 2016, we saw a significant increase in mobile game installs. China was a powerhouse in the first quarter eclipsing the US in number of game installs. However, the popularity of Android and iOS devices showed significant variations by region. Among the top 10 countries in terms of total game installs, Brazil and the Republic of Korea are the countries with the highest percentage of game installs on Android, while Japan has the highest share of iOS installs.

Android continues to lead global game installs, but iOS users upgrade to the latest version more rapidly. While most iOS users are on version 9.0 (released September 2015),  the most popular Android versions are still Kitkat (4.4) and Lollipop (5.0 and 5.1).

Beyond the platforms, what does the device landscape look like? Samsung holds an overwhelming lead globally. However, China is an exception with strong local brands, led by Xiaomi. Apple customers prefer their games on iPhones compared to iPads. Even more interesting, though,  is the fact  that people appear to be holding on to older versions of their iPads — the opposite trend of iPhones.

In Q1 2016, around 220,000 Unity-made games generated 4.2 billion installs across 1.7B unique devices globally. That’s a hefty amount of data, and now we’re making the insights and trends that it reveals available to you. Enjoy!

Download Your Copy Now

More Information:

April 25

Xamarin Evolve 5k

There's been a tradition for the more energetic attendees to do a morning 5k run during Xamarin Evolve - and 2016 will be no different! Xamarins have been out training most mornings, and we look forward to meeting and running with our customers. It's a FUN run, not a race - if you've got your gear and can complete the distance, please join us! Follow @conceptdev for twitter updates.

Date: Wednesday 27th (first day of conference, before the keynote)
Time: 6:15am
Place: outside Hyatt reception (near the flagpoles)

The Course

We'll run a flat 5km (3.2 mile) loop anti-clockwise around the conference center and surrounds:


It'll be early morning so traffic should be light (based on the past few days), however there are a couple of road crossings and safety will be the priority over speed. This is what the meeting place looks like at 6:15am:


For the speedsters, hang around at the end if possible to celebrate with everyone, get a group photo, and ensure you get mini-hack credit for completing the run!


Debugging memory corruption: who the hell writes “2” into my stack?!

Hi, my name is Tautvydas and I’m a software developer at Unity working in the Windows team. I’d like to share a story of debugging an elusive memory corruption bug.

Several weeks ago we received a bug report from a customer that said their game was crashing when using IL2CPP scripting backend. QA verified the bug and assigned it to me for fixing. The project was quite big (although far from the largest ones); it took 40 minutes to build on my machine. The instructions on the bug report said: “Play the game for 5-10 minutes until it crashes”. Sure enough, after following instructions, I observed a crash. I fired up WinDbg ready to nail it down. Unfortunately, the stack trace was bogus:

0:049> k
# Child-SP RetAddr Call Site
00 00000022`e25feb10 00000000`00000010 0x00007ffa`00000102

0:050> u 0x00007ffa`00000102 L10
00007ffa`00000102 ?? ???
^ Memory access error in ‘u 0x00007ffa`00000102 l10’

Clearly, it tried executing an invalid memory address. Although the stacktrace had been corrupted, I was hoping that only a part of the whole stack got corrupted and that I should be able to reconstruct it if I look at memory contents past the stack pointer register. Surely enough, that gave me an idea where to look next:

0:049> dps @rsp L200
……………
00000022`e25febd8 00007ffa`b1fdc65c ucrtbased!heap_alloc_dbg+0x1c [d:\th\minkernel\crts\ucrt\src\appcrt\heap\debug_heap.cpp @ 447]
00000022`e25febe0 00000000`00000004
00000022`e25febe8 00000022`00000001
00000022`e25febf0 00000022`00000000
00000022`e25febf8 00000000`00000000
00000022`e25fec00 00000022`e25fec30
00000022`e25fec08 00007ffa`99b3d3ab UnityPlayer!std::_Vector_alloc<std::_Vec_base_types<il2cpp::os::PollRequest,std::allocator<il2cpp::os::PollRequest> > >::_Get_data+0x2b [ c:\program files (x86)\microsoft visual studio 14.0\vc\include\vector @ 642]
00000022`e25fec10 00000022`e25ff458
00000022`e25fec18 cccccccc`cccccccc
00000022`e25fec20 cccccccc`cccccccc
00000022`e25fec28 00007ffa`b1fdf54c ucrtbased!_calloc_dbg+0x6c [d:\th\minkernel\crts\ucrt\src\appcrt\heap\debug_heap.cpp @ 511]
00000022`e25fec30 00000000`00000010
00000022`e25fec38 00007ffa`00000001
……………
00000022`e25fec58 00000000`00000010
00000022`e25fec60 00000022`e25feca0
00000022`e25fec68 00007ffa`b1fdb69e ucrtbased!calloc+0x2e [d:\th\minkernel\crts\ucrt\src\appcrt\heap\calloc.cpp @ 25]
00000022`e25fec70 00000000`00000001
00000022`e25fec78 00000000`00000010
00000022`e25fec80 cccccccc`00000001
00000022`e25fec88 00000000`00000000
00000022`e25fec90 00000022`00000000
00000022`e25fec98 cccccccc`cccccccc
00000022`e25feca0 00000022`e25ff3f0
00000022`e25feca8 00007ffa`99b3b646 UnityPlayer!il2cpp::os::SocketImpl::Poll+0x66 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\win32\socketimpl.cpp @ 1429]
00000022`e25fecb0 00000000`00000001
00000022`e25fecb8 00000000`00000010
……………
00000022`e25ff3f0 00000022`e25ff420
00000022`e25ff3f8 00007ffa`99c1caf4 UnityPlayer!il2cpp::os::Socket::Poll+0x44 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\socket.cpp @ 324]
00000022`e25ff400 00000022`e25ff458
00000022`e25ff408 cccccccc`ffffffff
00000022`e25ff410 00000022`e25ff5b4
00000022`e25ff418 00000022`e25ff594
00000022`e25ff420 00000022`e25ff7e0
00000022`e25ff428 00007ffa`99b585f8 UnityPlayer!il2cpp::vm::SocketPollingThread::RunLoop+0x268 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\vm\threadpool.cpp @ 452]
00000022`e25ff430 00000022`e25ff458
00000022`e25ff438 00000000`ffffffff
……………
00000022`e25ff7d8 00000022`e25ff6b8
00000022`e25ff7e0 00000022`e25ff870
00000022`e25ff7e8 00007ffa`99b58d2c UnityPlayer!il2cpp::vm::SocketPollingThreadEntryPoint+0xec [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\vm\threadpool.cpp @ 524]
00000022`e25ff7f0 00007ffa`9da83610 UnityPlayer!il2cpp::vm::g_SocketPollingThread
00000022`e25ff7f8 00007ffa`99b57700 UnityPlayer!il2cpp::vm::FreeThreadHandle [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\vm\threadpool.cpp @ 488]
00000022`e25ff800 00000000`0000106c
00000022`e25ff808 cccccccc`cccccccc
00000022`e25ff810 00007ffa`9da83610 UnityPlayer!il2cpp::vm::g_SocketPollingThread
00000022`e25ff818 000001c4`1705f5c0
00000022`e25ff820 cccccccc`0000106c
……………
00000022`e25ff860 00005eaa`e9a6af86
00000022`e25ff868 cccccccc`cccccccc
00000022`e25ff870 00000022`e25ff8d0
00000022`e25ff878 00007ffa`99c63b52 UnityPlayer!il2cpp::os::Thread::RunWrapper+0xd2 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\thread.cpp @ 106]
00000022`e25ff880 00007ffa`9da83610 UnityPlayer!il2cpp::vm::g_SocketPollingThread
00000022`e25ff888 00000000`00000018
00000022`e25ff890 cccccccc`cccccccc
……………
00000022`e25ff8a8 000001c4`15508c90
00000022`e25ff8b0 cccccccc`00000002
00000022`e25ff8b8 00007ffa`99b58c40 UnityPlayer!il2cpp::vm::SocketPollingThreadEntryPoint [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\vm\threadpool.cpp @ 494]
00000022`e25ff8c0 00007ffa`9da83610 UnityPlayer!il2cpp::vm::g_SocketPollingThread
00000022`e25ff8c8 000001c4`155a5890
00000022`e25ff8d0 00000022`e25ff920
00000022`e25ff8d8 00007ffa`99c19a14 UnityPlayer!il2cpp::os::ThreadStartWrapper+0x54 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\win32\threadimpl.cpp @ 31]
00000022`e25ff8e0 000001c4`155a5890
……………
00000022`e25ff900 cccccccc`cccccccc
00000022`e25ff908 00007ffa`99c63a80 UnityPlayer!il2cpp::os::Thread::RunWrapper [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\thread.cpp @ 80]
00000022`e25ff910 000001c4`155a5890
……………
00000022`e25ff940 000001c4`1e0801b0
00000022`e25ff948 00007ffa`e6858102 KERNEL32!BaseThreadInitThunk+0x22
00000022`e25ff950 000001c4`1e0801b0
00000022`e25ff958 00000000`00000000
00000022`e25ff960 00000000`00000000
00000022`e25ff968 00000000`00000000
00000022`e25ff970 00007ffa`99c199c0 UnityPlayer!il2cpp::os::ThreadStartWrapper [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\win32\threadimpl.cpp @ 26]
00000022`e25ff978 00007ffa`e926c5b4 ntdll!RtlUserThreadStart+0x34
00000022`e25ff980 00007ffa`e68580e0 KERNEL32!BaseThreadInitThunk

Here’s a rough reconstructed stacktrace:

00000022`e25febd8 00007ffa`b1fdc65c ucrtbased!heap_alloc_dbg+0x1c […\appcrt\heap\debug_heap.cpp @ 447]
00000022`e25fec28 00007ffa`b1fdf54c ucrtbased!_calloc_dbg+0x6c […\appcrt\heap\debug_heap.cpp @ 511]
00000022`e25fec68 00007ffa`b1fdb69e ucrtbased!calloc+0x2e […\appcrt\heap\calloc.cpp @ 25]
00000022`e25feca8 00007ffa`99b3b646 UnityPlayer!il2cpp::os::SocketImpl::Poll+0x66 […\libil2cpp\os\win32\socketimpl.cpp @ 1429]
00000022`e25ff3f8 00007ffa`99c1caf4 UnityPlayer!il2cpp::os::Socket::Poll+0x44 […\libil2cpp\os\socket.cpp @ 324]
00000022`e25ff428 00007ffa`99b585f8 UnityPlayer!il2cpp::vm::SocketPollingThread::RunLoop+0x268 […\libil2cpp\vm\threadpool.cpp @ 452]
00000022`e25ff7e8 00007ffa`99b58d2c UnityPlayer!il2cpp::vm::SocketPollingThreadEntryPoint+0xec […\libil2cpp\vm\threadpool.cpp @ 524]
00000022`e25ff878 00007ffa`99c63b52 UnityPlayer!il2cpp::os::Thread::RunWrapper+0xd2 […\libil2cpp\os\thread.cpp @ 106]
00000022`e25ff8d8 00007ffa`99c19a14 UnityPlayer!il2cpp::os::ThreadStartWrapper+0x54 […\libil2cpp\os\win32\threadimpl.cpp @ 31]
00000022`e25ff948 00007ffa`e6858102 KERNEL32!BaseThreadInitThunk+0x22
00000022`e25ff978 00007ffa`e926c5b4 ntdll!RtlUserThreadStart+0x34

Alright, so now I knew which thread was crashing: it was the IL2CPP runtime socket polling thread. Its responsibility is tell other threads when their sockets are ready to send or receive data. It goes like this: there’s a FIFO queue that socket poll requests get put in by other threads, the socket polling thread then dequeues these requests one by one, calls select() function and when select() returns a result, it queues a callback that was in the original request to the thread pool.

So somebody is corrupting the stack badly. In order to narrow the search, I decided to put “stack sentinels” on most stack frames in that thread. Here’s how my stack sentinel was defined:

Stack sentinel

When it’s constructed, it would fill the buffer with “0xDD”. When it’s destructed, it would check if those values did not change. This worked incredibly well: the game was no longer crashing! It was asserting instead:

Somebody wrote 2

Somebody had been touching my sentinel’s privates – and it definitely wasn’t a friend. I ran this a couple more times, and the result was the same: every time a value of “2” was written to the buffer first. Looking at the memory view, I noticed that what I saw was familiar:

Memory view

These are the exact same values that we’ve seen in the very first corrupted stack trace. I realized that whatever caused the crash earlier was also responsible for corrupting the stack sentinel. At first, I thought that this was some kind of a buffer overflow, and somebody was writing outside of their local variable bounds. So I started placing these stack sentinels much more aggressively: before almost every function call that the thread made. However, the corruptions seemed to happen at random times, and I wasn’t able to find what was causing them using this method.

I knew that memory was always getting corrupted while one of my sentinels is in scope. I somehow needed to catch the thing that corrupts it red handed. I figured to make the stack sentinel memory read only for the duration of the stack sentinel life: I would call VirtualProtect() in the constructor to mark pages read only, and call it again in the destructor to make them writable:

Protected sentinel

To my surprise, it was still being corrupted! And the message in the debug log was:

Memory was corrupted at 0xd046ffeea8. It was readonly when it got corrupted.
CrashingGame.exe has triggered a breakpoint.

This was a red flag to me. Somebody had been corrupting memory either while the memory was read only, or just before I set it to read only. Since I got no access violations, I assumed that it was the latter so I changed the code to check whether memory contents changed right after setting my magic values:

Checking right after setting

My theory checked out:

Memory was corrupted at 0x79b3bfea78.
CrashingGame.exe has triggered a breakpoint.

At this point I was thinking: “Well, it must be another thread corrupting my stack. It MUST be. Right? RIGHT?”. The only way I knew how to proceed in investigating this was to use data (memory) breakpoints to catch the offender. Unfortunately, on x86 you can watch only four memory locations at a time, that means I can monitor 32 bytes at most, while the area that had been getting corrupted was 16 KB. I somehow needed to figure out where to set the breakpoints. I started observing corruption patterns. At first, it seemed that they are random, but that was merely an illusion due to the nature of ASLR: every time I restarted the game, it would place the stack at random memory address, so the place of corruption naturally differed. However, when I realized this, I stopped restarting the game every time memory became corrupted and just continued execution. This led me to discover that the corrupted memory address was always constant for a given debugging session. In other words, once it had been corrupted once, it would always get corrupted at the exact same memory address as long as I don’t terminate the program:

Memory was corrupted at 0x90445febd8.
CrashingGame.exe has triggered a breakpoint.
Memory was corrupted at 0x90445febd8.
CrashingGame.exe has triggered a breakpoint.

I set a data breakpoint on that memory address and watched as it kept breaking whenever I set it to a magic value of 0xDD. I figured, this was going to take a while, but Visual Studio actually allows me to set a condition on that breakpoint: to only break if the value of that memory address is 2:

Conditional data breakpoint

A minute later, this breakpoint finally hit. I arrived at this point in time after 3 days into debugging this thing. This was going to be my triumph. “I finally pinned you down!”, I proclaimed. Or so I so optimistically thought:

Corrupted at assignment

I watched at the debugger with disbelief as my mind got filled with more questions than answers: “What? How is this even possible? Am I going crazy?”. I decided to look at the disassembly:

Corrupted at assignment disassembly

Sure enough, it was modifying that memory location. But it was writing 0xDD to it, not 0x02! After looking at the memory window, the whole region was already corrupted:

rax memory

As I was ready to bang my head against the wall, I called my coworker and asked him to look whether I had missed something obvious. We reviewed the debugging code together and we couldn’t find anything that could even remotely cause such weirdness. I then took a step back and tried imagining what could possibly be causing the debugger to break thinking that code set the value to “2”. I came up with the following hypothetical chain of events:

1. mov byte ptr [rax], 0DDh modifies the memory location, CPU breaks execution to let debugger inspect the program state
2. Memory gets corrupted by something
3. Debugger inspects the memory address, finds “2” inside and thinks that’s what changed.

So… what can change memory contents while the program is frozen by a debugger? As far as I know, that’s possible in 2 scenarios: it’s either another process doing it, or it’s the OS kernel. To investigate either of these, a conventional debugger will not work. Enter kernel debugging land.

Surprisingly, setting up kernel debugging is extremely easy on Windows. You’ll need 2 machines: the one debugger will run on, and the one you’ll debug. Open up elevated command prompt on the machine which you’re going to be debugging, and type this:

Enable kernel debugger

Host IP is the IP address of the machine that has the debugger running. It will use the specified port for the debugger connection. It can be anywhere between 49152 and 65535. After hitting enter on the second command, it will tell you a secret key (truncated in the picture) which acts as a password when you connect the debugger. After completing these steps, reboot.

On the other computer, open up WinDbg, click on File -> Kernel Debug and enter port and key.

Attaching kernel debugger

If everything goes well, you’ll be able to break execution by pressing Debug -> Break. If that works, the “debugee” computer will freeze. Enter “g” to continue execution.

I started up the game and waited for it to break once so I could find out the address at which memory gets corrupted:

Memory was corrupted at 0x49d05fedd8.
CrashingGame.exe has triggered a breakpoint.

Alright, now that I knew the address where to set a data breakpoint, I had to configure my kernel debugger to actually set it:

kd> !process 0 0
PROCESS ffffe00167228080
SessionId: 1 Cid: 26b8 Peb: 49cceca000 ParentCid: 03d8
DirBase: 1ae5e3000 ObjectTable: ffffc00186220d80 HandleCount:
Image: CrashingGame.exe
kd> .process /i ffffe00167228080
You need to continue execution (press ‘g’ ) for the context
to be switched. When the debugger breaks in again, you will be in
the new process context.
kd> g
Break instruction exception – code 80000003 (first chance)
nt!DbgBreakPointWithStatus:
fffff801`7534beb0 cc int 3
kd> .process
Implicit process is now ffffe001`66e9e080
kd> .reload /f
kd> ba w 1 0x00000049D05FEDD8 “.if (@@c++(*(char*)0x00000049D05FEDD8 == 2)) { k } .else { gc }”

After some time, the breakpoint actually hit…

# Child-SP RetAddr Call Site
00 ffffd000`23c1e980 fffff801`7527dc64 nt!IopCompleteRequest+0xef
01 ffffd000`23c1ea70 fffff801`75349953 nt!KiDeliverApc+0x134
02 ffffd000`23c1eb00 00007ffd`7e08b4bd nt!KiApcInterrupt+0xc3
03 00000049`d05fad50 cccccccc`cccccccc UnityPlayer!StackSentinel::StackSentinel+0x4d […\libil2cpp\utils\memory.cpp @ 21]

Alright, so what’s going on here?! The sentinel is happily setting its magic values, then there’s a hardware interrupt, which then calls some completion routine, and that writes “2” into my stack. Wow. Okay, for some reason Windows kernel is corrupting my memory. But why?

At first, I thought that this has to be us calling some Windows API and passing it invalid arguments. So I went through all the socket polling thread code again, and found that the only system call that we’ve been calling there was the select() function. I went to MSDN, and spent an hour rereading the docs on select() and rechecking whether we were doing everything correctly. As far as I could tell, there wasn’t really much you could do wrong with it, and there definitely wasn’t a place in docs where it said “if you pass it this parameter, we’ll write 2 into your stack”. It seemed like we were doing everything right.

After running out of things to try, I decided to step into the select() function with a debugger, step through its disassembly and figure out how it works. It took me a few hours, but I managed to do it. It seems that the select() function is a wrapper for the WSPSelect(), which roughly looks like this:

auto completionEvent = TlsGetValue(MSAFD_SockTlsSlot);
/* setting up some state

*/
IO_STATUS_BLOCK statusBlock;
auto result = NtDeviceIoControlFile(networkDeviceHandle, completionEvent, nullptr, nullptr, &statusBlock, 0x12024,
buffer, bufferLength, buffer, bufferLength);
if (result == STATUS_PENDING)
WaitForSingleObjectEx(completionEvent, INFINITE, TRUE);
/* convert result and return it

*/

The important part here is the call to NtDeviceIoControlFile(), the fact that it passes its local variable statusBlock as an out parameter, and finally the fact that it waits for the event to be signalled using an alertable wait. So far so good: it calls a kernel function, which returns STATUS_PENDING if it cannot complete the request immediately. In that case, WSPSelect() waits until the event is set. Once NtDeviceIoControlFile() is done, it writes the result to statusBlock variable and then sets the event. The wait completes and then WSPSelect() returns.

IO_STATUS_BLOCK struct looks like this:

typedef struct _IO_STATUS_BLOCK
{
union
{
NTSTATUS Status;
PVOID Pointer;
};
ULONG_PTR Information;
} IO_STATUS_BLOCK, *PIO_STATUS_BLOCK;

On 64-bit, that struct is 16 bytes long. It caught my attention that this struct seems to match my memory corruption pattern: the first 4 bytes get corrupted (NTSTATUS is 4 bytes long), then 4 bytes get skipped (padding/space for PVOID) and finally 8 more get corrupted. If that was indeed what was being written to my memory, then the first four bytes would contain the result status. The first 4 corruption bytes were always 0x00000102. And that happens to be the error code for… STATUS_TIMEOUT! That would be a sound theory, if only WSPSelect() didn’t wait for NtDeviceIOControlFile() to complete. But it did.

After figuring out how the select() function worked, I decided to look at the big picture on how socket polling thread worked. And then it hit me like a ton of bricks.

When another thread pushes a socket to be processed by the socket polling thread, the socket polling thread calls select() on that function. Since select() is a blocking call, when another socket is pushed to the socket polling thread queue it has to somehow interrupt select() so the new socket gets processed ASAP. How does one interrupt select() function? Apparently, we used QueueUserAPC() to execute asynchronous procedure while select() was blocked… and threw an exception out of it! That unrolled the stack, had us execute some more code, and then at some point in the future the kernel would complete the work and write the result to statusBlock local variable (which no longer existed at that point in time). If it happened to hit a return address on the stack, we’d crash.

The fix was pretty straightforward: instead of using QueueUserAPC(), we now create a loopback socket to which we send a byte any time we need to interrupt select(). This path has been used on POSIX platforms for quite a while, and is now used on Windows too. The fix for this bug shipped in Unity 5.3.4p1.

This is one of those bugs that keep you up at night. It took me 5 days to solve, and it’s probably one of the hardest bugs I ever had to look into and fix. Lesson learnt, folks: do not throw exceptions out of asynchronous procedures if you’re inside a system call!

April 20

Particle System Modules – FAQ

Starting with Unity 5.3, you have full scripting access to the particle system’s modules. We noticed these new scripting features can be a bit confusing. Why do we use structs in an unconventional manner?

In this blog post, we will answer some of your questions, take a little peek behind the scenes and explain some of our plans to make the process more intuitive in the future.

Accessing Modules

An example

Here is a typical example in which we modify the rate property from the Emission module.

using UnityEngine;

public class AccessingAParticleSystemModule : MonoBehaviour
{
  // Use this for initialization
  void Start ()
  {
    // Get the emission module.
    var forceModule = GetComponent<ParticleSystem>().forceOverLifetime;

    // Enable it and set a value
    forceModule.enabled = true;
    forceModule.x = new ParticleSystem.MinMaxCurve(15.0f);
  }
}

To anyone familiar with .NET, you may notice that we grab the struct and set its value, but never assign it back to the particle system. How can the particle system ever know about this change, what kind of magic is this!?

It’s just an interface

Particle System modules in Unity are entirely contained within the C++ side of the engine. When a call is made to a particle system or one of its modules, it simply calls down to the C++ side.
Internally, particle system modules are properties of a particle system. They are not independent and we never swap the owners or share them between systems. So whilst it’s possible in script to grab a module and pass it around in script, that module will always belong to the same system.

To help clarify this, let’s take a look at the previous example in detail.
When the system receives a request for the emission module, the engine creates a new EmissionModule struct and passes the owning particle system as its one and only parameter. We do this because the particle system is required in order to access the modules properties.


public sealed class ParticleSystem : Component
{
.....

  public EmissionModule emission { get { return new EmissionModule(this); } }

....
}

public partial struct EmissionModule
{
  private ParticleSystem m_ParticleSystem; // Direct access to the particle system that owns this module.

  EmissionModule(ParticleSystem particleSystem)
  {
    m_ParticleSystem = particleSystem;
  }

  public MinMaxCurve rate
  {
    set
    {
      // In here we call down to the c++ side and perform what amounts to this:
      m_ParticleSystem->GetEmissionModule()->SetRate(value);
    }
  }
......
}

When we set the rate, the variable m_ParticleSystem is used to access the module and set it directly. Therefore we never need to reassign the module to the particle system, because it is always part of it and any changes are applied immediately. So all a module does is call into the system that owns it, the module struct itself is just an interface into the particle systems internals.
Internally, the modules store their respective properties and they also hold state based information, which is why it is not possible for modules to be shared or assigned to different particle systems.
If you do want to copy properties from one system to another, it is advised that you only copy the relevant values and not the entire class, as this results in less data being passed between the C++ side and the C# side.

The MinMaxCurve

A significant number of module properties are driven by our MinMaxCurve class. It is used to describe a change of a value with time. There are 4 possible modes supported; here is a brief guide on how to use each when scripting.

Constant

By far the simplest mode, constant just uses a single constant value. This value will not change over time. If you wanted to drive a property via script, then setting the scalar would be one way to do this.

1In script, we would access the constant like this:

using UnityEngine;

public class MinMaxCurveConstantMode : MonoBehaviour
{
  ParticleSystem myParticleSystem;
  ParticleSystem.EmissionModule emissionModule;

  void Start()
  {
    // Get the system and the emission module.
    myParticleSystem = GetComponent<ParticleSystem>();
    emissionModule = myParticleSystem.emission;

    GetValue();
    SetValue();
  }

  void GetValue()
  {
    print("The constant value is " + emissionModule.rate.constantMax);
  }

  void SetValue()
  {
    emissionModule.rate = new ParticleSystem.MinMaxCurve(10.0f);
  }
}

Random between two constants

This mode generates a random value between two constants. Internally, we store the two constants as a key in the min and max curves respectively. We get our value by performing a lerp between the 2 values using a normalised random parameter as our lerp amount.This means that we are almost doing the same amount of work as the random between two curves mode.

1

This is how we would access the two constants of a module:

using UnityEngine;

public class ParticleModuleExamples : MonoBehaviour
{
  ParticleSystem myParticleSystem;
  ParticleSystem.EmissionModule emissionModule;

  void Start()
  {
    // Get the system and the emission module.
    myParticleSystem = GetComponent<ParticleSystem>();
    emissionModule = myParticleSystem.emission;

    GetRandomConstantValues();
    SetRandomConstantValues();
  }

  void GetRandomConstantValues()
  {
    print(string.Format("The constant values are: min {0} max {1}.", emissionModule.rate.constantMin, emissionModule.rate.constantMax));
  }

  void SetRandomConstantValues()
  {
    // Assign the curve to the emission module
    emissionModule.rate =new ParticleSystem.MinMaxCurve(0.0f, 1.0f);
  }
}

Curve

In this mode, the value of the property will change based on querying a curve. When using curves from the MinMaxCurve in script there are some caveats.
Firstly, if you try to read the curve property when it is in one of the Curve modes, then the you’ll get the following error message: “Reading particle curves from script is unsupported unless they are in constant mode”.
Due to the way the curves are compressed in the engine, it is not possible to get the MinMaxCurve unless it is using one of the 2 constant modes.We know this isn’t great, read on our plans to improve it. The reason for this is that internally, we don’t actually just store an AnimationCurve, but select one of two paths. If the curve is simple (no more than 3 keys with one at each end), then we use an optimized polynomial curve, which provides improved performance. We then fallback to using the standard non-optimized curve if it is more complex. In the Inspector, an unoptimized curve will have a small icon at the bottom right which will offer to optimize the curve for you.

An optimized curve

1

An unoptimized curve

1

While it may not be possible to get the curve from a module in script, it is possible to work around this by storing your own curve and applying this to the module when required, like this:

using UnityEngine;

public class MinMaxCurveCurveMode : MonoBehaviour
{
  ParticleSystem myParticleSystem;
  ParticleSystem.EmissionModule emissionModule;

  // We can "scale" the curve with this value. It gets multiplied by the curve.
  public float scalar = 1.0f;

  AnimationCurve ourCurve;

  void Start()
  {
    // Get the system and the emission module.
    myParticleSystem = GetComponent<ParticleSystem>();
    emissionModule = myParticleSystem.emission;

    // A simple linear curve.
    ourCurve = new AnimationCurve();
    ourCurve.AddKey(0.0f, 0.0f);
    ourCurve.AddKey(1.0f, 1.0f);

    // Apply the curve
    emissionModule.rate = new ParticleSystem.MinMaxCurve(scalar, ourCurve);

    // In 5 seconds we will modify the curve.
    Invoke("ModifyCurve", 5.0f);
  }

  void ModifyCurve()
  {
    // Add a key to the current curve.
    ourCurve.AddKey(0.5f, 0.0f);

   // Apply the changed curve
   emissionModule.rate = new ParticleSystem.MinMaxCurve(scalar, ourCurve);
  }
}

Random between 2 curves.

This mode generates random values from in between the min and a max curves, using time to determine where on the x axis to sample. The shaded area represents the potential values. This mode is similar to the curve mode in that it is not possible to access the curves from script and that we also use optimized polynomial curves(when possible). In order to benefit from this, both curves must be optimizable, that is contain no more than 3 keys and have one at each end. Like in curve mode, it is possible to tell if the curves are optimized by examining the bottom right of the editor window.

1

This example is very similar to the curve, however, we now also set the minimum curve as well.


using UnityEngine;

public class MinMaxCurveRandom2CurvesMode : MonoBehaviour
{
  ParticleSystem myParticleSystem;
  ParticleSystem.EmissionModule emissionModule;

  AnimationCurve ourCurveMin;
  AnimationCurve ourCurveMax;

  // We can "scale" the curves with this value. It gets multiplied by the curves.
  public float scalar = 1.0f;

  void Start()
  {
    // Get the system and the emission module.
    myParticleSystem = GetComponent<ParticleSystem>();
    emissionModule = myParticleSystem.emission;

    // A horizontal straight line at value 1
    ourCurveMin = new AnimationCurve();
    ourCurveMin.AddKey(0.0f, 1.0f);
    ourCurveMin.AddKey(1.0f, 1.0f);

    // A horizontal straight line at value 0.5
    ourCurveMax = new AnimationCurve();
    ourCurveMax.AddKey(0.0f, 0.5f);
    ourCurveMax.AddKey(1.0f, 0.5f);

    // Apply the curves
    emissionModule.rate = new ParticleSystem.MinMaxCurve(scalar, ourCurveMin, ourCurveMax);

    // In 5 seconds we will modify the curve.
    Invoke("ModifyCurve", 5.0f);
  }

  void ModifyCurve()
  {
    // Create a "pinch" point.
    ourCurveMin.AddKey(0.5f, 0.7f);
    ourCurveMax.AddKey(0.5f, 0.6f);

    // Apply the changed curve
    emissionModule.rate = new ParticleSystem.MinMaxCurve(scalar, ourCurveMin, ourCurveMax);
  }
}

Performance

We did some simple performance comparisons to see how these different modes compare. These samples were taken before our recent SIMD optimizations, which should provide a significant performance boost. In our specific test scene, we got these results:
1

Easing the pain

Reading curves from the MinMaxCurve class

We know that you really ought to be able to read particle system curves from script, regardless of what mode they are in. We are actively working on removing this limitation, so you can read/modify all those lovely curves in your scripts. Its also currently not possible to query the curve to check if the mode is using a curve, without an error being thrown. That’s also going to be fixed!

Changing modules from structs to classes

We are currently prototyping a change to move all the structs to classes. Functionally they will behave the same, however by using a reference type, it should be clearer that the module belongs to a system. It will also allow for setting/getting values without having to hold a temporary variable. However, this will mean we allocate them once in the constructor, which will generate garbage, but only at initialization time.

For example:

var em = ps.emission;
em.enabled = true;

// Could be written as

ps.emission.enabled = true;

Finally

We hope this post has been of some use to you, please head over to this forum post to grab the examples and comment/discuss.
We will also add this information over to our documentation.

Best practices for rewarded video ads

A recent study, “In-Game Advertising the Right Way: Monetize, Engage, Retain” which emerged from a survey of over 2,000 mobile game developers and players, takes an in-depth look at rewarded video ads’ potential to help developers monetize and drive user engagement. The infographic below summarizes the findings.

You can download the full study here.

Video_Ads_Infographic_r4_A

Video_Ads_Infographic_r4_B

Video_Ads_Infographic_r4_C

Video_Ads_Infographic_r4_D

Video_Ads_Infographic_r4_E

Video_Ads_Infographic_r4_F

Monologue

Monologue is a window into the world, work, and lives of the community members and developers that make up the Mono Project, which is a free cross-platform development environment used primarily on Linux.

If you would rather follow Monologue using a newsreader, we provide the following feed:

RSS 2.0 Feed

Monologue is powered by Mono and the Monologue software.

Bloggers