da Bomb - Badge Post-Mortem

Intro

Welcome to our second badge post-mortem. John here, Team Lead, Team Ides.

We’re going to talk today about we did right, everything we did wrong, and how we think things went. In general, “da bomb” was one of the most successful projects we’ve ever done and we’re proud to say that our planning worked.

There’s a bomb-shaped hole in our hearts now that the project is done, and we’re eagerly working towards next year. This document is a way of capturing what we learned and is a work in progress and we will update it as we remember our triumphs and failures.

So, how’d it go?

Planning

First off, we had a huge head start on the build because we hadn’t done a badge for DC26. Personal issues and other problems got in the way.

In that time, Bill had done an enormous amount of work porting ChibiOS to the BMD340, saving us from that issue this year.

I feel like overall, planning was very good. The schedule we posted during the Kickstarter Process was fairly spot-on, give or take a few weeks.  We paced ourselves against the DC25 schedule we’d previously accomplished and while at some points the DC27 build lagged behind the DC25 work, we were producing 500 vs. 225 badges, and eventually we went faster than the DC25 build.

We budgeted about two months of slip into the schedule and used up nearly all of that time! The major problems contributing to slips in our schedule were:

• I had the wrong URL on the boxes and stickers, had to buy new stickers to cover up the mistakes, and lost a week to relabelling boxes.
• Macrofab’s supply of CREE LEDs ran out, forcing a last minute parts change.
• Macrofab had a near ten day delay getting boards back to us because their PCB supplier had delays.
• That slip-up caused us to lose a weekend, one in which we had planned to have about ten people come over and solder the through-hole components. It's hard to get people to come over during weekdays/weeknights.
• It actually took four straight days at about eight hours a day of soldering to finish 500 boards. Next time we get robots!

Income and Expenses

The total badge cost was $45,433.98, not including rooms, transportation, and defcon badges for the team.  All-In, costs get a bit closer to $53,117.16.

We raised $39,426.55, putting us over budget by $13,750.61 (fuck.).

I had to float that on credit cards, and that sort of sucked. Another $1,100 off to Sixt Rental agency for a Jeep Wrangler for the week and we were off to DEF CON.  Sixt was very easy to work with and I’ll use them again.

A single board's BOM (Bill-Of-Materials) clocked in around $93. When doing the pricing, I forgot to include ancillary expenses like prototyping, bank fees, and shipping from our vendors. The mistake we made here is that we were paying an additional $15.37 per badge, bringing the "profit" margin to around $11 badge. Margins were really that slim.

Pricing

We sold the badges between $120 and $150 depending on where you bought and how you paid. Pre at $120, Cash at about $130, Stripe/Square sales at $140 to cover the 2.5%+0.25 fee.

Interestingly enough, people at DEF CON were willing to swipe/dip their credit cards into my square reader. The trust is that strong in the Square brand now I guess.

At times, some revenue had to go to to vendors and other middle men like credit card fees. 30% is a pretty universal margin on most things sold in the US, and we feel that we were in a good price range (but should have gone higher – the market would have easily bared higher prices given the limited availability of our badges and functionality.

We were probably the 1st or 2nd highest priced badge but we feel that we offered a lot of art and functions, so it was okay.

After moving so much money around to multiple continents and vendors,  we maybe, just maybe squeaked out a profit of about $195.09 at my best estimates. That's #badgelife! Given that there's badges left, and we had to ship out a replacement or two, we're technically still in the hole. :(

Like last time we bought a tiny bit of equipment to make things go faster. A new Hakko FR-931 soldering iron was acquired and heavily used.  

Back in DC25 we bought a new Oscilloscope, AmScope microscope, and Hakko hot air station. Those purchases saved the entire project.

We used Square for in-person sales (the $32 reader was a good value), and Stripe for online sales with custom code. It worked out very well.  I only wish I could have done all sales on one merchant account. It’s not easy.

Hacker Warehouse was also a huge help, selling around 100 of our boards in the vendor area. So, a special thanks go to ggee for putting up with us, and another big thank you to our sponsor, New Context, we couldn’t have done this without them.

Limited failures and an excellent yield of 99.4%!

Yield was incredibly good this year, so good we couldn’t believe it. Out of 500 boards, 496 boards were functional with the remaining boards having only minor issues.

• One board had an unresponsive CPU. Sigh.
• One board had a dead LED controller chip which, try as hard as I could with the hot air station, I couldn’t fix because the PCB trace came off and I was out of magnet wire.
• A number of boards had badly soldered joysticks, but this is more do to the craptacular design of the joystick and less to do with the fabrication job. Let’s not use those joysticks again.
• As of today we have 25 fully working kits in inventory ($3500 @ $140 each) to be sold, 5 with bad caps and/or LEDs, and 2 donor boards. This equates to a loss of about $644.00 in broken inventory. I think we're going to put these up on Tindie soon and try to recover some of our lossage.

Overall the build and packing out of Macrofab was very good. Macrofab has come a long, long way and our design skills have improved in the last few years. Did I mention I finally built a completely silent audio circuit in stereo? Yeah!

Hardware mistakes

• Prototyping: I think we lost some time to the fragile wiring of our prototypes. We'd attached an incredible number of wires to the nRF52840-dev boards and I probably snapped off one or two of them a night while moving the boards. It's best to get your PCB prototypes as soon as you can to avoid this kind of mess, but prototypes are expensive, right? Ours were $280 each, and we spent nearly $800 before we had working boards – two spins, one with one board, one with two.
• Caps: I really wish I hadn’t used large electrolytic caps on the board, on the back, in the audio circuit. They would break off and delaminate the traces, rendering that portion of the board useless.  I’m going to buy some copper tape and repair these boards for resale, just not right now. I broke three badges, around my neck, thanks to the wrecking-ball style official DC27 badge.
• SAO connectors: The SAO connectors were too close to the screen, making some SAOs incompatible. We offered ( 2 x 3 x 2.54mm ) extenders to anyone who asked to fix this. They were also too close to a couple of LEDs and on one board, the pressure caused from the SAO connector next to an LED caused the LED to snap off, and you guessed it, take the copper trace with it. Because the LEDs are arranged in a grid, the now disconnected line would take out a big chunk of LEDs with it.
• SAO GPIO2 to GND: Shorting GPIO2 to ground started the New Context PIN Entry Puzzle. I wish we hadn’t allowed this to happen because it sure made debugging of bad SAOs hard. It was also stupid easy to short that pin with another badge, lanyard clip, or random part.
• SAO I2C BUS: The SAO i2C bus was shared with LED Controller. We should have had a dedicated SAO bus, and we did not. Next time, we exile the SAOs to their own bus. This issue caused AND!XOR’s Doom badge to be EEPROM-erased when we started up comms to our LED Controller Chip. What a mess that was. Bill and the AND!XOR Team sorted this out with some last minute commits and there are plenty of comments in our github repo if you want to read about that.
• Power MOSFET/charging circuit: I neglected to include a power MOSFET which would disconnect the battery and charging circuit from the rest of the system.
  
  This mistake caused system power to go crazy as the charging controller chip tried to regulate the entire system voltage. We had to make this parts change last minute, we could not test it (we were only able to simulate it in SPICE), and we could not see it working on a prototype. Still, we pushed on and ordered 500 boards with this change. We were confident that the change would work, as it was simulated and verified against reference designs. It was still a very, very, scary, nail-biting last minute change. If it had failed we would've had to rework 500 boards by hand.
  
  The lesson here is read everything before you design. I'd never even seen a a copy of Microchip's "AN1149: Designing a Li-Ion Load Sharing System" document until after I'd designed the charging and power circuits! Had I seen that, I would have had a correct design, but alas, I did not.
  
  Another good reference for this sort of thing is the Li-Ion System Power Path Management Reference Design. Read those before you start building charging and power supplies on your badge. No really, read them.  
  
• Silkscreen / Cosmetics: We should not have done a RED silkscreen with a matte black PCB. The face of our bomb-dude was impossible to see. Inverting this, by covering a RED board with BLACK silkscreen would have been amazing looking, but sadly, we didn't get this right.

Software

Some stats:

• 1,100+ commits to our repo, spanning over 22 months of work.
• 5,500+ lines of code to make the sea battle game go.
• 27 fully functional applications
• Custom, low-level drivers for over 15 on-board devices. (go Bill!)

Let's have a massive round of applause for Bill. Bill kicked serious ass at debugging, firmware, drivers, and all things in general. Couldn’t have done this without him!

I have to say that throughout the project the one thing I really didn’t want to face was the game. It was going to be very hard to write (real-time, multiplayer code isn’t easy), involve a technology I hadn’t worked with before (Bluetooth Low Energy) and a multitude of new game problems. Once I started coding it, it seemed easy, but the sprite library and inability to have enough memory to double-buffer the screen made for some fun times. Don't be like me. Get a MCPU with enough RAM.  But hey, on to the game...

Game Design

We're pretty married to uGFX on our badges. We know it well, and can make it do things that it probably shouldn't be able to do like sprites and overlay.

The game was created by taking the state machine from the DC25 Roman Fighting game that we made and then designing around that engine. I wrote a very basic rigid physics engine (all in C) that took into account things like drag,  acceleration, ship size, and motion. A bit of code later and we had the world map done. Later, we threw out my early code base and replaced it with Devon's sprite engine.

Once I had a basic motion system, Egan drew a map and ships, we wrote code to generate bullets, collisions, and motion. We wrote all of this by hand in C with very little in the way of external libraries or support aside from uGFX. It was grueling.

I really want to be able to use PyGame or something else on our next badge and worry more about the game and less about the game's engine.

It took weeks to get a game that Egan could come by and tune. That is, we had to have something playable before we could work on the rules, and we didn't have a lot of time to refine them. Egan and I had quite a few late nights tuning the game, and Devon came in a few times to assist us with the sprite library he wrote.  It solved a lot of problems for us like working in limited memory and partial updating of graphics, which made things run a lot smoother.

The game is fairly fluid, but we did ship with some bugs, so let's talk about that...

Software Mistakes

There’s still a couple of unfortunate sprite glitches in the game. If you go to the far right on the screen, it will incorrectly draw the ship, and I haven’t had time to find this dumb bug. We’ll get to it eventually!

Everything else worked fine, and the extra Flash+RAM on the BMD340 made life real easy.

Did you know? There’s nearly 4GB left on the SD card and 600KB of flash for you to play in. Reformat the SD card. We shipped a 4GB partition over to our copying company, and all they had left was 8GB SD Cards, so you got 4GB for free, matey! Go write a game or something!

We had a last minute, show stopping network bug -- One thing I did not understand is how the Nordic soft device actually sent data. You put data into a buffer in memory, tell the soft device to send it, and then you wait until you get an acknowledgement event. This all happens asynchronously.

My original software was loading data into the same buffer and changing the buffer over and over again. This would cause the packets to be clobbered, and the game to end. Bill and I worked out a last minute ring-buffer fix that made the game go smoothly.

We’d still like to find a CPU with enough memory that we can safely double-buffer a large screen. Our badge for next year might use that, if we can find something we like.

Shipping and Pickups, or, why you shouldn’t do pickups.

Shipping is surprisingly expensive. Around $1,300 of the project went to shipping. We charged people $15, but that was including packing materials, box, and USPS, so let's say we made maybe $1 on shipping there.

We kitted and shipped 160 badges, two weeks before DEF CON by USPS Priority mail. Average shipping cost was around $7.95, and we shocked the USPS lady when we walked in with a massive cart of priority mail!

All orders made it to their destination with the exception of one which was lost in the mail (USPS said it was signed for and picked up, we’re still filing a claim for our $100 or so)

We then did three pickup windows at DEF CON. I never want to do pickups again.

They are too confusing, and when 100 people descend on your hotel room, and you’ve got a crew of six people kitting, testing, and putting badges in people’s hands right then and there it gets pretty hectic.

A couple of times we marked I accidentally marked an order as shipped when it wasn’t because of issues with Shipstation and poor network response in Bally’s (thanks DEFCON!). This required some apologizing, but eventually we sorted all of the mismarked orders out.

ProTip: It's incredibly important to have a paper copy of your shipping / pickup manifest when you are at DEF CON. The network might go away. We found that practically no one lied about their badge pickup status, but we had a hard time keeping track. We double-tracked orders on shipstation and on paper.

Currently, we're still sorting out about 15 or so shipments which people haven't picked up or called about after we sent multiple emails! arrgh!

We also spent two 8-hour shifts in the HHV waiting for people to show up. Rarely did they show up after day one, which took away a lot of valuable DEF CON time!

Shipping really is the way to go. You can resolve all of the issues before the con at your leisure. This also means keeping at least three weeks between your ship date and the event, or you won’t have time to ship and fix things.

We created SKU’s for each product type -- either you got the challenge coin and the sticker sheet and some extra goodies, or you did not -- and then started to import data.

We were extremely reliant on shipstation.com, which is part of stamps.com. We were able to import backer reports from Kickstarter, orders from Stripe, and manual orders with simple python scripts. Those CSV files were then used with shipstation’s label maker to create single-sheet packing list and shipping label sheets which we could use to ship our orders. Combined with the free bags and obscene amounts of bubble wrap from Macrofab’s shipment, we were able to ship easily.

Things for Next time

I’d like to only ship. No pickups at all. I feel like I'd be much saner.

I’d like to not use Kickstarter to raise money next year. They take too much in fees, and we can probably do this on our own, on our own online store.

I'd like to focus on making a custom case next year and more artwork on the PCB.

After two years, I think we’re capable of doing this ourselves, but Kickstarter lends a certain amount of trust/caché that random people trust. We’ll see how it goes next time.

Thanks!

Thank you for supporting this crazy thing. It’s such a pleasure to see people using our hardware and enjoying our work. It’s been a fun ride building this with you, and we hope to start a new badge project in a few months. We just need to sleep first ;)

Special Thanks

To Corie, Charles, James, Jon, Jeri, Josh, Seth, and everyone else who came over, soldered, and put things into boxes. To our friends and significant others who had to deal with us during the build process, and to everyone for contributing and making the project happen.

To Whitney, and the entire [redacted] Slack where we make things happen behind the scenes.

To the many friends in #badgelife, including those that allowed us to group-by batteries (Cmdc0de),  CPUs (and!Xor), and SAO connectors (TwinkieTwinkie).

See you next year.