On What Went Wrong at the End- More Reflections from ECTC11

The number one question that I've seen come out of Event Camp Twin Cites regarding the technical side of things is, "Dude, what happened at the end?" For those that did not see it, there was an almost comic meltdown of the Skype connections to the Pods.  A kind and well written summary from Mitchell Beer can be found Here.

Some of this is conjecture, as we had to tear down and vacate the venue in very short order, so further testing could not be done.  What follows is a rough compilation of the many things that contributed to not only the bizarre ending to ECTC11, but the Skype problems in general throughout the conference.

It has been asked, rightfully so, why didn't we test all of this before going live.  I can tell you that as far as we were concerned we did.  They tested the lines, they tested the calls to all the pods, we tested the inputs, we tested the outputs, we tested the video inputs, we tested the video outputs.  It's a valuable lesson in something we all know- that there's no such thing as too much testing, or taking those test too far.  Sometimes it's just not enough.

All of the following contributed and played off each other, and unfortunately it is the interplay that caused the most serious problems- most of which would not have shown up in anything other than full scale, live testing, with the actual participants in the actual rooms with the actual equipment.  And probably the correct alignment of Mercury thrown in just for good measure.   ECTC, in essence became the full size test.  At least it's an environment that's theoretically set up for that purpose...

So here it is, to the best of my ability:

1) The number of pods- Last year there were two pods.  In true Event Camp Twin Cities fashion, they pushed the envelope and tried to have 9.  Eventually that number reduced to 7.  Because of the number of pods, especially the original 9, it seemed impractical to have nine dedicated machines, and we decided to try the group calling feature of Skype and had 4 pods on one computer, and 3 on another.  So having so many pods is why we started combining them on machines, which leads to...

2) Combining Pods 1- Combining the pods created a lot of noise on each of the two Skype machines.  Instead of one person at a time, you now had bunches of people talking, waving, saying hi, and I think that Skype was clamping down on some feeds to "promote" others.  It's certainly the way it sounded in the headphones of the audio rig.  The wrong pods were being brought to the front of the mix.  It would make sense that Skype is geared towards what it's generally considered use is- chat between one or more individuals.  When individuals are chatting, we tend to wait our turn.  The noise from some pods seemed to be canceling other pods out, much like a Google Hangout tries to "decide" who's talking, and that can be overridden by someone typing to loudly.  To make matters worse, there was the problem in #4, but we'll get to that in a second.  All of this would be fixable if only we had the ability to somehow mute the audio of some of the pods when one was speaking, which leads us to...

3) Combining pods 2-  According to the Podmaster (as I desperately probed around for a solution to stop the madness), there was no way to mute individual pods on Skype.  I do not know this for sure, as I don't personally have the premium version of Skype with the multi-person chat.  What I do know is that the recent redesign of the interface for Skype is a bloody mess, and if there were controls to mute the audio, good luck finding them.  You're more likely to accidently bring up and call your Aunt Judy trying to figure out the right combination of hidden rollovers and hieroglyphs.

It should be noted at this point, that in a further attempt to salvage the segment, we hung up on all the pods and tried calling a couple of them one at a time.  When we knocked it down to a single call to Amsterdam, though, their audio feed was clearly being cut in and out by the noise limiter on Skype.   I am again not familiar enough with Skype to know if there's a setting that could have been changed on their end, but it was again very obvious when listening via headphones.  It may have been possible to overcome with some time, perhaps by having someone come closer to the mic on the computer and by having all other hush, but before we got to that point I was told we had Silicon Valley on the line on the other machine.  When we connected on a single call to Silicon Valley, Mike appeared to be on a headset, and it sounded awesome.  I plugged and unplugged the audio jacks on the Mac so I could talk to him- the drawback of routing the audio signal through the house was that we didn't have a good talkback method, and we were all set to go back to him.  Unfortunately, though, we just plain ran out of time.  We had a hard out at 2pm, Kurt was wrapping things up in the room, and the decision was made to scrap it.

4) Pod instructions/Combining Pods 3- (sensing a common thread?) Despite meticulous instructions, and without throwing anyone under the bus, it seemed like every time we went to a Skype machine, at least one of the three (or four) would have their audio turned up on the Sonic Foundry feed.  This contributed to the confusion, and exacerbated problems 2 and 3 because we couldn't mute them.  People still weren't listening to the right feed, and the delay ate us alive.  Furthermore, the audio in the room then contains the potted-up Skype audio, which contained the audio of the delayed webcast feed, which is now being sent back to the other pods.  Yeesh...

5) Panic.  I regret having to put this one in, but it's true.  When things go south, your mind is racing, and you try everything you can think of.  Sometimes, though, the moment passes and it just wasn't enough.  You don't think of a solution until the next day.  Or the next week.  It's like that great comeback for an insult that you don't think of until the jerk's walked away.

I can't imagine what it was like up there for Sam, and he kept his cool very well.  The best description I heard was that he was the straight guy in a comedy routine that he didn't know he was in.  My suggestion in perfect 20-20 hindsight, however, is that when the first one wasn't answering, we needed to just stop and wait to see how long it took them to respond.  Discover their delay and deal with it.   Some have suggested some kind of in-room clock or audio cue in dealing with delayed audiences, as continuing to speak (while a perfectly natural reaction) only adds to the confusion.

If an actual 30 seconds went by, which I know is an ETERNITY, then we'd know that something was wrong beyond them just being on the wrong feed.  As it was, it felt like Sam would move on in what seemed to me to be less than 20 seconds, and then we're suddenly being answered by the Pod he'd  just left.  He'd try and go back to that one, only to be answered by the one he'd just moved on to.

So that's what happened.  Feel free to pick it apart and tell me what I may have missed.  And if you know where it is, for the love of God please share where the mute button is in Skype.

Otherwise, my recommendations coming out of this are:

1) Reduce the number of Pods if at all possible to 4, and put them all on their own machines.  4 inputs is where the lowest levels of video switchers tend to hang around, so you can have a dedicated switch just for flipping between Pods for not a lot of money.  That switch then sends its signal to whatever your main video switcher is.  If you need to scale up, scale at that point and get a bigger Skype switcher, but I really feel like 1-1 machines might be imperative to making this all work.

2) Maybe to reduce noise, perhaps you give "voice" to the leaders of the pods and give them a headset?  Just spitballing... It might overcome the limitations of combining pods.

3) If the machines are separated, your audio feeds will need to be separated, so again your going to need more channels on your audio mixer, or a completely separate mixer for the Skype machines.  Either way, it gets you individual control over the audio feeds, and you can mute whoever's mixing margaritas in the background.

4) Did any of the pods notice they were being fed the main video feed instead of looking out the I-Sight cameras in the MacBooks?  What's your feedback on the video quality, other than any buffering or obvious Skype-related things?  I'm still experimenting, and if I figure it out I'll share.  We may try it again at Event Camp Europe.  Suffice it to say that it's remarkably low tech and inexpensive, and I think could be a really nice key to making this all work.

5) It should go without saying, but I will.  When it comes to trying new tech, try and emulate the final use scenario as closely as possible during testing.  We thought we had, but clearly there were factors at work that we didn't anticipate.  At least now you know to...