Wednesday, October 24, 2012

Windows 8 Doesn’t Want Your App. Try Again Later

[Update 3: And it finally passed certification and is now in the store!  (Oddly enough though, I didn't actually change any code between the last release and this one.)  Thanks to everyone that checked this post out and commented here or elsewhere, and thanks to the people at Microsoft that helped me through a lot of the issues.  While I wrote this to point out the flaws in the overall certification process, and ways it could improve, the individual people that I worked with were always wiling to go out of their way to try and help.  As mentioned below, they already had this escalated it internally before this was posted.]

[Update 2: A lot of people are commenting on various sites about how it is my fault that my app failed, as it was due to problems with my code.  I agree completely, but the point of the article is not that Microsoft failed my app, but that they don't clearly explain why they failed it, which turned me failing once or maybe twice to failing 6 times.]

[Update: I just went to the Microsoft Store this morning to pickup a Surface tablet (so I could test on an ARM computer), and after getting my pre-order slip over the weekend, and being told that they open early at 8, I show up at 7 and am first in line!  At 7:30 though, I overhear that Microsoft sent out an email yesterday saying that they couldn't open the store until their normal time of 10.  I have a job, so I left Surfaceless and un-happy.  #NotHowToLaunchAMajorProduct]

To start, let’s get one thing straight: Microsoft desperately needs apps in its store.  If not only for Windows 8, especially for Windows RT, which can only run apps from there.  It also wants as many apps in there as possible, and has been very active in courting and helping developers to write app for it.  For the most part, I think they have done a great job, as the tools, documentation and examples are excellent.  Where they really fall down though is in the last mile: app submission.

I first submitted my app, Memorylage, to the store on August 29.  Since then, it has failed validation 6 times, and I still don’t know what is causing the failure.  Below, I talk about what I have gone through in the past 10 months, trying to get my app ready, but since it is a little long winded, I have a “Short Version” further down if you want to skip to there. (I won’t blame you)  Also, please subscribe to my blog if you find it interesting.

The Long Version

Getting Ready For Submission

Memorylage has been in development since December, back when there was only the Developer Preview to use.  It was sorely lacking documentation and examples back then, but was enough to get familiar with what is a very different development experience from the Windows of old.  By Consumer Preview, things had improved greatly, especially in the templates, and Memorylage was starting to take shape.

Right around the release of the Release Preview, I attended a 3 day App Excellence lab that Microsoft hosted in their Waltham, MA office.  As this was just a hobby, I had use up my vacation time to go, but it was a great way to get some focused development done, and meet some other developers.  I also won the App X contest at the end, netting me a new Lumia 800 Windows Phone.  Best of all though, I was able to schedule time with a Microsoft Field Engineer, flying in from Redmond, to review my app for early access to the store.

The review involved a 60-point checklist we went though, which checked to make sure that the app met all aspects of their design guidelines, and really showcased how a Windows 8 app was supposed to work.  He also dug deep into the performance and stability of the app, helping me find ways to improve the responsiveness, and give me ideas about how to improve it.  There ended up being 3 issues for me to fix before he would grant me early access (2 of which I knew about before the meeting, but wanted input on), but a few weeks later (July 12th), I had resolved the issues and was granted an early access developer token.

While I had the token, allowing me to submit to the store before almost anyone else, I really wanted to make sure the app was polished and feature complete before submitting, so I kept working on it to get it ready.  I then went to a Hackathon, hosted by Microsoft at the Museum of Science in Boston, on August 18th where I won my second App X competition, netting me a Samsung Series 7 Slate, a $500 gift card and an XBox game.  I also met plenty of people from Microsoft and elsewhere who were really interested in my app, and excited to see it in the store.  They encouraged me to just hurry up and submit it, so I was really feeling good about the possibility of it taking off.

I made up a list of things I felt I need to have ready before submitting, and quickly worked through them, eventually submitting to the store on August 29th.  With only 500 apps in the store, and almost 2 months before general release of Windows 8, I was feeling pretty good about the work I had put into it.  A day later though, I got the first of what would be many failure reports.

Where Things Fall Apart

Memorylage failed a day later for 4 reasons, and with the exception of the 1st, there really wasn’t much insight into what the issue was.
  • Requirement 1.2: The website linked in the app and listing page was not finished
  • Requirement 3.2: The app crashes
  • Requirement 3.8: Performance
  • Requirement 3.10: Failed the Direct 3D test
I was surprised that they actually looked at the web site (I was planning on having it up by the time it made it to the store), but was happy to see that level of detailed review, as I figured it reflected well on the level of end-to-end quality that they were looking for. For the other failures though, I was left with quite a few questions.
  • When does the app crash?
    • At startup?
    • During a certain task?
    • Every time, or randomly?
  • What performance problems are there?
    • Does it startup too slow?
    • Does it suspend too slow?
    • Does a certain action hang or take too long?
  • How did my app fail the Direct 3D test?
    • It is a C#/XAML app that only uses standard controls, and should not have any DirectX requirements.
    • It was submitted as a neutral app, and accepted all DirectX levels
  • Why did none of these failures occur when running the WACK (Windows App Certification Kit) on my own computer?

All of these are pretty basic questions that a developer would need to know to reliably be able to fix the issues that the store is finding. Staying positive though, I took this as an opportunity to really spend a few days doing nothing but polish work on the app. I went through every section and feature, cleaning up and speeding up anything I wasn’t 100% happy with. In the end, I felt confident that the app was truly store-ready, and that my app was better for the whole ordeal. That is, until it failed again for the same reasons! (minus the web site issue)

Bringing In Some Help

After trying a third time, releasing only for x86 (thinking there could be an issue of ARM devices, which developers weren’t able to test at that point), I couldn’t think of anything else to do with the vague results I was getting.  So, I reached out to Bob, a developer evangelist that I met at the Hackathon at the Museum of Science.  He asked me to send him my appx package, so that he and others could look at it themselves. and to post my questions in the support forums.  I did both, and got many suggestions from various people, but everyone was basically just guessing, and telling me to try something and re-submit.  Since access to the store was open by now though, each submission took longer and longer to come back.

One person from the forums who worked at Microsoft actually had his own test store, and asked me to send him my app to run through certification there.  I was excited to have someone looking closely at my app, who could give me a more detailed explanation of what was causing the failure, but no such luck.  The app passed in his store, giving me no insight into the failures in the real store.

test_store

I was then given a link to Microsoft's support site (Windows store developers get two free questions that will be answered by someone at Microsoft, so definitely check it out) and while the person I originally talked to couldn't give me any answers, he escalated the issue to someone on the store team.  It took a few days, but he did get back to me through email, and wrote the following:

Hello Jeffrey, 

I've reviewed this and found out what the concern was. I need you to resubmit the app to make sure that the issue has been resolved. I will be following the process of this submission so that I can verify at which stage the failure should pass. If it does not I will be following up with you to resolve. 

Thank you,
-John

Without any indication as to what the issue actually was, he seems to be saying that it was something on their end, so I resubmitted the app, like he said.

After 6 days of staring at this…

Content Compliance

…it fails with the same three errors.

Making Progress

I eventually get to talk to John on the phone.  He tells me about the little information he can get from others on the store team, and even from some of the server logs used in testing.  He says that it is failing on an ARM machine and a low-powered test computer, due to it crashing, and that because it crashes, the other two tests fail as well (which doesn't make sense to me).  He still can't explain why the app appears to pass all tests in the technical compliance phase, but then fails in the content compliance phase though.  His best advice to me is to try and test on the slowest computer possible, see if I can find any issues, and if that doesn’t work, just re-submit so that he can talk to certain people if it fails again.

Store Failure Report

With the amount of time it took last time to simply fail again, I wanted to try one more time to find out what the issue was before re-submitting.  I was already testing on a slow netbook, as well as an even slower VM, but apparently that wasn't enough.  I then went to my VM settings and crippled it even more.  I lowered to RAM to 1GB, slowed the CPU to roughly 1Ghz, and turned off 2D and 3D acceleration.  Running the app didn't turn up anything new though, as it launched with no issues, and quickly as well.  Thinking that there wasn't anything else I could do, I created a new app package to submit, and ran the WACK (Windows Application Certification Kit) one last time when… BINGO!  It finally failed a test on my computer!

Closer, but Not Quite There

Memorylage failed the WACK for taking 2.46 seconds to suspend, with a limit of only 2.  It also stated that the app used 1.5MB of IO during suspend, which was important, since my suspend code only saves a single string (the standard code from Microsoft for saving navigation history), meaning that there must have been contention trying to suspend while loading a folder.  With this information, I had a fix in under a half hour, but for some reason the store kept hiding it from me in the error reports, keeping me from finding it.

Since I was also had an un-easy feeling that the Microsoft Ad SDK I was using was causing the Direct 3D failure, I took that out as well and re-submitted.  Both the performance and Direct 3D tests did indeed pass this time, but it was still crashing on startup.

While I was heartened to see that 2 of the 3 failures were cleared up, that crashing failure (which I had NEVER seen myself) was starting to feel like it would keep me from ever getting into the store.  After a few days of moping around, I read an article that mentioned that the app store had over 3,600 apps, and Windows 8 was only 3 weeks away from general release.  This pushed to me take one final, and thorough look through ALL of my startup code.

I literally yelled (scaring my wife) when I found the issue staring at me in the very first place I looked.

In the constructor of App.xaml.cs was a call to CurrentAppSimulator.LicenseInformation that was left over from some initial test code I had written (all license code in use was using the correct object).  Since the constructor normally never has any real code in it, and is usually collapsed, I had never looked before to see that it was left.  Remembering that I had read about how calling that object will cause your app to crash when released in the store (but not when using it outside of the store) I knew that this had to be what was causing it to crash in the store every time.  After a little more checking to be sure there wasn’t anything else, I re-submitted to the store, and waited 10 days until… it failed again!

[Edit: I know that it was my fault for missing this line, but it would have been nice if it was caught by the WACK, or at least if the error report told me I missed it.  It was actually listed in the top 5 reasons for failures in the store, so I am not the only one.  Regardless, even after fixing it, it still failed.]

Another Valiant Attempt at Escalation

Ever since my third failure, I had been told by about 3-4 different people that my issue was being escalated internally, and that it should be resolved soon.  This never actually yielded any results for me, but after emailing with Allan yet again (who has done everything he could to try and help), said that it would be escalated to his boss’ boss’ boss’ boss (a corporate VP).  Needless to say, my hopes didn’t get too high.

Conveniently though, I got the failure notice only a few hours before going to a live recording of Dot Net Rocks (probably the most popular podcast on developing with Microsoft’s technologies) on the 17th.  I got there early, and got to meet the hosts, Carl Franklin and Richard Campbell, and even show them my application (I brought the Samsung Slate I had won) and told them about what I had gone through so far.  The guys were awesome, and really sympathetic about my issues, and actually quite mad about how Microsoft was handling things with the store. They said they would try and find someone that could help me out (their international tour was sponsored by Microsoft, and they constantly interview people from the company, so they have connections), but as this was just another promise of escalation, I kept my hopes low.

I emailed them the next day though, just to follow up, and thank them for taking the time to talk to me.  Richard emailed me back the next day saying that he yelled at one of the developer evangelists in New York about what developers are going through, and that hopefully the message would make its way to Steve Sinofsky (the president of the entire Windows division).  He then said that there are thousands of apps stuck in the approval process, which is why it is so backed up.  I didn’t think one more evangelist hearing about it would change anything though (I already had three tying to help me), and thought that would be it.  A few minutes later though, I got another email from Richard saying that he thought about it again, and decided to email Sinofsky himself.

This was definitely more than just a random “escalation”, so I started to get a little more hopeful, and Sinofsky even replied the next morning, saying that they would get back to me directly.  This definitely got me hopeful, but yet again, I seemed to have been let down, as I have not heard anything more from him or anyone else about it. (In Sinofsky’s defense though, he does have a bit of an important launch in a couple days to focus on).


One Last Ongoing Hope

Allan has been keeping me up to date on what he has been doing on his end, and he just had me re-submit my app today, and forwarding me an email from someone saying that they would prioritize my app when it was back in submission.  I still don’t know what was causing the last failure, and I haven’t heard back since, so I doubt I will make it in before launch, but I could be wrong.

Either way though, nothing can the fact that the current approval process is hurting developers, and that is something that Microsoft cannot afford right now.  I have already seen some progress as I have gone through my various submissions, and I am sure that in 6-12 months, none of these issues will exist, but I don’t think Microsoft can afford to wait that long.


The Short Version

What I Did to Get Ready For Submission

  • I started working on Memorylage back in December, with the Developer Preview.  I have been programming for Windows for 17 years, with C# for the last 6, so I wasn’t starting from scratch
  • I read through all of the their design and coding documentation, trying to really create something that showed off Windows 8’s features, and its design style
  • I re-wrote the app for the Consumer Preview when it was released, since a lot had changed, and the new version of Visual Studio wouldn’t even open up files from the Developer Preview anyways
  • I attended a 3-day App Excellence lab at Microsoft’s Waltham MA office, where Memorylage won the App X challenge at the end of it
  • I also setup a meeting with a Microsoft Field Engineer to review my app, so that I could get early access to the store.  This process was extremely detailed, with a 60+ point checklist of criteria that my app needed to meet to get it.  I had three things that I needed to fix when I first met him, but I passed the second time
  • I continued working on the app without actually submitting yet, because I wanted to make sure that it was fully polished and feature complete
  • I signed up for Microsoft’s 30-To-Launch program, which helps developers go through all of the steps necessary to get into the store.  I already had gone through everything that they showed, but I figured it couldn’t hurt anyways, in case there was something new
  • I went to a Microsoft hosted Hackathon at the Museum of Science in Boston, where Memorylage again won the App X challenge at the end.  This time, I netted a $1,200 Samsung Series 7 Slate with dock, a $500 gift card and an XBox game
  • I met several Microsoft people (and others) there that were encouraging me to submit my app to the store immediately, since they thought it was ready, and was something they wanted in the store
  • After another two weeks of polishing, I finally submitted my app on August 29th (almost a full two months before the release of Windows 8)

What Happened After Submission

  • The app failed for 4 reasons, 3 of which gave me no real idea of what caused the failure.  The failures were for stopping responding (when did this happen?), performance (performance of what?), Direct 3D (I don’t use Direct 3D) and having an incomplete web site (I had it finished a day later)
  • I went through all of my code, cleaning up performance anywhere I could, and tried to fix anything that I wasn’t 100% happy with
  • I submitted again, and failed again for the same 3 reasons (minus the web site issue)
  • I emailed Bob, a developer evangelist who I met at the Boston Hackathon, who said that he would escalate the issue internally
  • I re-submitted again, but only for x86, thinking it may be an issue with ARM (since no developers had access to ARM to test with), but it still failed for the same reasons
  • I posted multiple thread in the support forums, asking for help with what I thought was an issue with the startup time (the performance failure) and the Direct 3D failure
  • Matt, who works at Microsoft and replied to the app startup post, asked me to send him my app, so he could test it in his own personal store.  It passed all tests for him though, so I didn’t find out anything new
  • I went to Microsoft’s support site, and used one of my two free help requests to talk to someone directly through chat.  They couldn’t help me, but escalated me to someone on the store team, who would get back to me via email
  • A few days later, I got an email from John, the person my support ticket was escalated to, saying that he thinks everything should be all set now, and to just re-submit.  He never said what he thought was wrong in the first place
  • I resubmit and fail again for the same reasons.  This time, it took 6 days to fail, since everyone had access to the store now.  Each submission was now becoming a lot more costly as the release of Windows 8 was starting to get closer
  • John tells me that the failure is because I submitted a neutral app, which will be tested against ARM, but that can’t be it, because I had already tested as x86 only (the third submission).  He also said that the Direct 3D failure was because I didn’t have a check against the feature level, and not because I used any Direct 3D features.  This was also incorrect, as the documentation clearly states that you only need to check the feature level if using Direct X 10 or up
  • I get to talk to John on the phone, and he tells me that they couldn’t find out much information about my failures (even after pulling server logs), except that the app was apparently crashing during startup on a low-powered machine
  • I cripple my VM, so that it is SUPER slow, and run the WACK (Windows Application Certification Kit), which finally causes it to fail.  The failure is due to taking too long to suspend, and the information it gives allows me to fix it in under a half hour.  Why the store hides this information, and just gives a generic “Performance” error is beyond me
  • I also remove the Microsoft Ad SDK, since I have a feeling that it might be causing the Direct 3D failure
  • I re-submit, and wait another 5 days for it to fail again.  The performance and Direct 3D issues were gone though, so it was just the crashing failure that was left
  • I find that my code has a leftover call to CurrentAppSimulator.LicenseInformation from initial development of the licensing code, and I know that this will cause the app to fail certification, but has no effect when running it on my own (making it very hard to catch) [Edit: I know that it was my fault for missing this line, but it would have been nice if it was caught by the WACK, or if the error report at least told me that I missed it.  It was actually listed in the top 5 reasons for failures in the store, so I am not the only one.  Regardless, even after fixing it, it still failed.]
  • I re-submit, and am hopeful that this will have fixed all of the issues, but after 10 days of waiting, it fails again with the same crashing issue
  • Allan (another developer evangelist that had been helping me a lot) says that he has escalated my issue to his boss’ boss’ boss’ boss (a corporate VP), and will still try to get it through before launch (this is now the 17th of October, 9 days until launch)
  • I meet Carl Franklin and Richard Campbell from Dot Net Rocks, at a live recording in Boston, who I got to show my app to, and tell about my issues.  They say they will try to get someone to help, and initially just talk to some developer evangelist that they meet in New York about it, but Richard then decides to email Steven Sinofsky himself (head of the entire Windows Division) about the issue
  • Also, while at the Dot Net Rocks event, I meet Chris, who ran the App Excellence lab I went to, and he let me test Memorylage on his ARM tablet, and it worked perfectly, so the issues were not ARM based
  • Sinofsky actually responds the next day, saying that they will get back to me directly (I never hear back though)
  • I attend a webcast from Microsoft about common pitfalls of submitting to the store.  The issue of calling the CurrentAppSimulator object is in the top 5 causes of failure, but thry still won’t catch that in any of the tests, or even tell you that was a cause for failure.  The developer is left to figure it out on their own
  • John asks me to send him my app again, so he can look into it again (I haven’t heard back from him yet)
  • Allan tells me that I am on the docket for some critical store meeting, so they are working on it
  • Allan forwards me an email, asking me to re-submit my app, and they will prioritize the certification process
  • I am still waiting to hear back on that (it is now just past midnight on the 25th, with just under 24 hours until launch), but that was only yesterday, so I am actually confident I will hear back soon
  • So far, I have sent/received 131 emails back and forth with various people within Microsoft
  • I have given my app, and even my source code, to multiple people within Microsoft.  To date, not a single one of them has been able to cause it to crash, or fail a single test.  Every issue that I have figured out, I found on my own, but the store still fails it.  How do you debug that?

Store Submission History

Conclusion

What Went Wrong

There are many reasons why my app is still not in the store, and many things that could be improved.  This is just a quick list of the ones I could think of.  Hopefully, people from the store team will see this, and work on improving things.  I have already noticed some improvement between the various submissions, but there is still a long way to go.
  • Failure reports need MORE INFORMATION.  Imagine hitting compile on a large program, and instead of the compiler saying that you missed a semi-colon, and on which line, it waits a week, and then just says that you have a syntax error.  That is what the certification process feels like
  • Do not hide information from the WACK.  If an app fails the suspend test, give the developer that information.  It already creates a detailed report, so just forward that along.
  • If a test fails, especially a performance related one, or a crashing issue, let the developer know what architecture it failed on.  As developers currently can’t test on ARM, letting them know that the app works great on x86 but fails on ARM would be really helpful.  Same with if it passes on a normal machine, but fails on a very low-powered one
  • I still have no idea why the Ad SDK causes the app to fail the Direct 3D test.  Everyone that I talked to says it shouldn’t, as it only uses a web viewer.  If it does have an issue though, then developers should be notified, so they aren’t left guessing
  • Check for use of CurrentAppSimulator in the WACK.  This was one of the top 5 reasons that apps failed validation, so it is a common mistake.  Having the WACK check for it allows developers to find it before they even submit.  At the very least though, tell the developer that it is the cause if it fails.  None of my error reports gave any hints about it
  • When an app fails in the Content Validation phase, where there is an actual person looking at it, they should spend a few minutes giving as detailed a response as possible.  Every response I got was a single sentence that told me nothing.  It may take them a little longer per submission, but if it keeps developers from re-submitting 7 times, it will speed things up overall, and keep developers happier

What I Lost Because of This

Even if my app does get into the store before launch, or very close to it, there are already plenty of things that I have lost, which can’t really be recovered.  Hopefully others have been luckier than me.
  • Almost 2 months of marketing for my app, with very little competition.  When I first submitted my app, there were only 500 apps.  Now there are over 7,000
  • Almost 2 months of development time.  I really couldn’t focus on adding new features when the current ones won’t pass.  When dealing with a new market place like this, development time is precious, and 2 months is a lot
  • Another Samsung Series 7 Slate.  The 30-To-Launch program is giving out a free slate to everyone who registered and got their app in the store before launch.  Even though I submitted 2 months early, with a completed and polished app, I lose out on $1,200 worth of hardware
  • Extra marketing from winning the Boston Hackathon.  Microsoft is offering to highlight Hackathon winners that have made it into the store, but the deadline in next Friday, and even if I make that, I have already lost the time over the past 2 months
  • Downloads and reviews.  If I had been in the store all this time, I would already have built up enough of a reputation to help bring my app higher in the search results, instead of starting from scratch.

In Conclusion

This post went on waaay longer than I planned so I will be brief.

I still think that Windows 8 is a great opportunity for developers, but as it stands, they are in for a world of hurt in trying to get through that last hurdle.  As a long-time Windows developer, I really hope that changes soon.

Also, I want to say that while the overall process may have been negative, every person that I talked to personally was always helpful, and willing to go out of their way to help me (especially Allan), and I don't want to put them in a negative light.  It is just that Microsoft is a large company, and despite their best efforts, things still turned out the way they have.

39 comments:

  1. Wow... that is very bad luck.

    We had the opposite experience. We ported a complex WP7 XNA game to Windows 8. We got invited to the App Excellence Lab. We won an early access token. We submitted to the store in July and passed on the first try. To date we've submitted to the store 3 times and passed all three times.

    Really weird that you've had this much trouble.

    ReplyDelete
    Replies
    1. especially as he writes " I was already testing on a slow netbook, as well as an even slower VM, but apparently that wasn't enough.".. so your complex WP7 XNA game works fine on the very reduced hardware requirements (well done). Makes me wonder if his code is very inefficient, or if the store has many different types of submission options.

      Delete
    2. This comment has been removed by the author.

      Delete
  2. I feel your pain. Have had this happen multiple times in the windows phone cert process. Multiple different testers getting different results etc.

    ReplyDelete
  3. "Microsoft desperately needs apps in its store."

    Your story has convinced me that the app store is a terrible idea. I will never buy a single app using it. I've read horror stories about the Apple store but as I'm not an Apple user I didn't really care. This is what happens when you allow someone to become a gatekeeper. No thank you.

    ReplyDelete
  4. What complex WP7 XNA was it?
    Hard to believe without a name.
    Jeff I would love to know if this app is on IOS, it sounds really good.

    ReplyDelete
  5. This may be relevant: http://pcper.com/reviews/Editorial/Windows-You-Love-Gone

    ReplyDelete
  6. I had a bit of a similar experience, but with Windows Phone. Quite frustrating and doesn't help Microsoft get developers on their side.

    http://www.johanpaul.com/blog/2012/09/wp75-backgroundaudioplayer-crashes/

    ReplyDelete
  7. We have another strange problem. Our app passed all certifications including updates. But says "Admin Suspended". We released update again assuming that some issues are there, but that also passed certification, but still not visible in store. There is no one to talk to about it, no one knows how to deal with it. We had to resubmit with a different name, and it passed again.

    ReplyDelete
  8. Delete the app form the store, upload it once again preferably with a new name. If there is nothing seriously wrong with your app, you'll get certified.

    Don't believe me? Here's my story.

    We had a similar situation. Our app failed certification 3 times for no particular reason...
    They said it was violating point 3.8 (suspension and start time). But it never occurred in our internal tests, we tested our app on a low end netbook (1.6Ghz, 2Gb, 5400 rpm).

    I got really frustrated and optimized the heck out of our app (HTML5 App Analyzer was very useful, but I believe there are no similar tools for XAML apps). In the meantime we decided to change the name of our app and we had to upload it as a totally new application because I couldn't change the app name in the Windows Store. And guess what... After one day our app got certified :-)

    ReplyDelete
  9. Thanks for telling your torrid tale, wishing you get all you deserve & more from M$!

    ReplyDelete
  10. Wow... thanks for sharing this story and exposing one of the biggest threats Microsoft has with Windows 8 at the moment. If they don't get developers on board, their platform is doomed.

    Being a developer currently struggling with getting an app certified myself, I certainly hope I won't have as much trouble as you - still I'm currently on day 37 past first submission, waiting for approval.

    ReplyDelete
  11. I went to 0 events, but made sure to read the documentation thoroughly. I checked all of my code for references to CurrentAppSimulator three times before submitting. Was certified and up on the app store my first attempt.
    Was complimented by Microsoft on my game quality and featured on the front of the store.

    ReplyDelete
    Replies
    1. Congratulations on getting in the store, and being featured. I have been reading the documentation and design guidelines carefully as well, for about 10 months. While I missed the CurrentAppSimulator until recently, even after removing it, it still fails. And the point isn't that my code is perfect, it's that they won't tell me what it is about my code that causes the failure, and not a single person in Microsoft has ever gotten it to crash or fail a test themselves.

      Delete
  12. This is a strong endorsement for the Android/open source model. I've submitted apps to the Apple App Store and Google Play market on the same day. The Apple store took 2 rejections and 3 months to approve. The Android app appeared in the Google Play market 10 seconds after I hit the button. Granted, they get a lot more spammers and junk. But, it seems easier to clean it up afterwards than to set up a giant barrier to entry which drives developers away. I hope you get your app in the Windows App store.

    ReplyDelete
  13. I think Bartosz Oczujda is correct.... They are way behind in there testing so rather than retesting the code after the first failure they are simply throwing it back over the fence by saying it failed again.

    ReplyDelete
  14. As you said, Microsoft is very open and helpful with developers these days. I hope your nice list of dotted improvements will make it into the process.

    ReplyDelete
  15. I actually had a hard time getting my app through as well. It got kicked out due to unspecified bugs. After talking with MS support via chat, they told me to re-submit and ask for a PDF of the bugs reports / failures.

    I did that and got one of the most useful bug reports ever from a QA team. I had two issues, one was some misleading marketing copy that implied certain functionality that the app actually didn't have.

    After that, it was a quick fix and a quick approval process.

    TLDR: Resubmit and ask for a detailed PDF bug report, otherwise you it will be frustrating and pointless.

    ReplyDelete
    Replies
    1. I have been getting PDF reports from the last 3 failures, and they have told me nothing. Apparently my app is crashing on their system every time (but has NEVER crashed for me or anyone at Microsoft) so the PDF is just a bunch of screen shots where the first is my splash screen, and the rest are all blank. I have asked in the notes to testers for more details, but have always been ignored.

      Delete
  16. why develop for windows anymore. didn't everyone stop doing that in the 90's?

    ReplyDelete
    Replies
    1. No, but it's looking like a lot may stop in the 2010s.

      (Captcha: "ntsucta"

      NT most definitely did not suck.

      Delete
  17. ...and I was hopeful to get certified in less than 3 days after my initial submission and be lucky enough to qualify for a slate :P

    My takeaway:

    - Explicitly check Direct 3D feature, no matter what.
    - Be aware of the Ad SDK
    - The CurrentAppSimulator.LicenseInformation issue
    - Run the WACK tests on a crippled VM

    Thanks for the extremely detailed post. It is even longer than my long posts but it is very clear and concise, give us a lot of insight on pitfalls.

    Best of luck to you and everyone else. I do hope that they fix this it is in **everyone's** best interest.

    ReplyDelete
  18. In response to the segment "What I lost because of this":

    First, let me mention that I also won the local Windows 8 Hackathon in Houston and received a Samsung slate and had some very stupid failures (using a support email with .co NOT .com extension for example), so I feel your pain.

    1. 2 months of marketing for my app - Most Win 8 users in the last 2 months were developers, some tech writers, and a few curious folk. The consumers haven't adopted it yet, so it's not that big of a loss. Just a bunch of devs 1-starring their competition right now.

    2. 2 months of development time - At the end of this two months you do have an app right? And you know what to expect going forward, so another way of viewing it is a 2 month jump start and a nice new line addition on your resume for a skill that is likely to begin increasing in demand any day now.

    3. Another Samsung Series 7 Slate - You mentioned you won a slate in a previous hackathon competition. It states in the terms and conditions for "The Great Slate Giveaway" on generationapp.com that "Recipients of Samsung Series 7 devices from previous or concurrent Microsoft Windows Store app offers are excluded from this offer. "

    4. Extra marketing from winning the Boston Hackathon - I was able to take advantage of this and it is very unlikely to have had much of an effect. They basically posted a picture of my app to a photo album on their facebook page along with a bunch of other mixed apps. Here is my 5 nanoseconds of fame: http://www.facebook.com/photo.php?fbid=479846988714502&set=pb.141819225850615.-2207520000.1351271059&type=3&theater

    5. Downloads and reviews - Again you aren't missing out much here, it's just a few users here and there, but mainly devs, and devs always give the most negative reviews. Seriously, devs will complain worse than any other demographic. For example, by look of your app it isn't much of an improvement over the built-in photos application. Get ready to have that pointed out to you more times than is necessary.

    ReplyDelete
    Replies
    1. Hey Paul, thanks for your comment. I mostly agree with all of your points about marketing, and I didn't know there was a limit on the slates. The biggest pain point for me though, is the loss of development time. While I may still be ahead of others, that is because I started in December. The fact remains that my app would have been better right now, with more features, if I didn't have to focus on certification for two months.

      In the long run though, you are right, and this will all just be a minor pain point of being an early adopter. Hopefully this post will help some of the current flaws get noticed and fixed faster though.

      Delete
  19. Great write up. This reminds me of the first Apple iPhone submissions, painful. Thanks again for all the details you've given!

    ReplyDelete
  20. I agree that the store process is not great in providing feedback - and in fact a couple of the problems you describe there sound nasty to trackdown without good information. However, these sorts of problems can be at least partly mitigated by good practice. Your VM sounds equal to systems I have to use, and expect to be running Win 8 in the wild... a machine below min spec with a ton of stuff running in the background will get you more value from the WACK test. Also, because it is non-deterministic running it several times will also help.

    Without proper care ARM devices might cut power to the CPU for instance - leaving your app running on some 300MHz equivalent core... 1GHz is a lot in that respect. Not to mention if you are using some intel PC your caches and buses are going to be lightning fast by comparison.

    The store simulator problem is a classic where, when implementing the feature you should make it togglable by preprocessor macro and put it in your settings for a release/master build so that you can't screw up... leaving code like that lying around instead of doing it properly is a classic gotcha.

    ReplyDelete
    Replies
    1. Thanks for the tips Semi! I will definitely be doing more testing with a crippled VM from now on, to stress the performance tests even more. As for the simulator issue, my code did properly handle the switch based on the build type, but the line that I found was left over from some initial testing, and all current code was in a different file. So it was just an issue of not properly cleaning up my test code. Obviously that's my fault, but it happens, and if they properly reported that to me, I would have failed once instead of 6 times (or even better, the WACK would have caught it before submission).

      Delete
  21. Neither the Apple App Store or the Microsoft App Store provide beta-testing or QA testing services. Developers working on software of all kinds really need to understand that crashes, UI issues and other problems have to be discovered and fixed prior to app submission. While no software can ever be perfect, developers nevertheless are not entitled to a pass on due diligence. Quite the contrary, a private app developer has to create his own team of beta testers and QA testers. Apple and Microsoft aren't staffing up to do detailed QA testing and reporting for private developers, nor should either company ever consider doing so.

    ReplyDelete
    Replies
    1. That's a non-sequitur. Nobody said they should.

      But if not even the automated tests are able to provide a better report that "it crashes", something is amiss. Or if you can re-submit the exact same app under a new name, and it passes.

      Even if you're doing manual testing (why? Google doesn't) on new apps, writing a ten word "what I did to get it to crash" sentence takes a negligible amount of time and saves both your company and your developers real man-hours=money in the long run.

      Delete
  22. Just got a call from the GenApp team, they wanted to check if as of today (Tuesday) my app was live yet. Just checked and it isn't.

    Lost my chance to a slate, submitted 9 days ago :'(

    What would make me awfully mad is that it would get returned at the last minute grrr

    ReplyDelete
    Replies
    1. I definitely understand the frustration with that. The best advice I would give would be to email them as soon as your app does get in the store, assuming that the current submission passes. If you can show them that you had a successful submission in the store over a week before the deadline, then you may be able to talk them into giving you a slate anyways. They really want to give those developers, as an incentive for them to develop more for Windows 8, and it couldn't hurt to ask.

      Delete
  23. I'm in the same boat. Failing 3.2, crashing. Store Team isn't helping. Support tickets opened not helping. Inside track with MSFT through my companies Account Manager isn't helping either.

    What, may I ask, did you do to reduce the Suspend time below the 2 sec. requirement?

    ReplyDelete
    Replies
    1. The only thing I was doing on suspend was the standard code from the Grid template, which saves the navigation history of the main Frame (which is just a single string). Since that only changes when switching pages, I just moved to the save code to the load of any page, and then commented out the code in the suspend event.

      Delete
  24. Your experience made me wonder if the app approval process is reason why the Store currently flooding with apps merely packaged static pages (like some so called fashion apps and make up advice apps). They got approved because these apps are ultra simple, no room to fail (because they require no internet connection, no storage, no nothing). More feature, more room of failure; No feature, no failure.

    This is bad.

    ReplyDelete
  25. yeah its even cheaper you can activate and purchase windows 8 pro for only $39 from http://www.windows8save.com

    I remember in 2009 how huge windows 7 was so really your blog makes sense.

    ReplyDelete
  26. It feels so nice to find somebody with some original thoughts on this subject. event staffing agencyReally thankful to you for starting this.

    ReplyDelete
  27. That’s a good comment and I feeling happy to read your post. I would like to request you please keep continue it.
    Windows 8 post

    ReplyDelete
  28. Thanks! For the interesting news. The way you present the news is just commendable but I like the most is information about windows apps which I found interesting.

    ReplyDelete