DevOps Zone is brought to you in partnership with:

Jon Archer is a software professional with over 15 years experience who is lucky enough to work out of his mountain retreat high above Denver in the Colorado Rockies. He has worked as a software engineer with various technologies during the course of his career as well as a couple of diversions managing teams. These days he is the scrum master for a challengingly distributed team with members in Massachusetts, Colorado and Hyderabad India. He is a passionate believer in agile principles and is a key advocate thereof for his current employer Perceptive Informatics. He blogs at and tweets as @9200feet Jon is a DZone MVB and is not an employee of DZone and has posted 19 posts at DZone. View Full User Profile

How to Completely Fail at BDD

  • submit to reddit

Are you interested in introducing BDD to your team? Don’t try and do it like this under these circumstances. Learn from my failure.

Having experienced ambiguous requirements specifications and inadequate testing of the applications I’ve worked on during my career as a software developer, when I saw BDD I thought I saw the future. In particular the Gherkin based implementations (CucumberSpecflow) with their Given/When/Then descriptions of scenarios for product features that could lead to executable specifications blew me away as full of potential.

I was incredibly eager to try this stuff out on a real project.

I played around a bit in my spare time with Cucumber. I learned how features were specified in plain text feature files and tests related to them were implemented in step definitions. You match the scenarios you describe in your feature files to step definitions that exercise your application code using regular expressions. I’d never really used regex that much so was initially a bit concerned that this would be a problem. It turned out though that with a small grab bag of regex tricks you could do most things you needed.

One of my first real problems I ran into was that I was looking to try BDD on an upcoming C# project. That put me in a minority group. I don’t entirely pretend to understand why, but much of the C# ecosystem seems pretty closed off, lacking in the use and acceptance of rich open source projects that I was used to in my prior life working on projects on the Java stack. (As an aside, I even notice differences when I interview C# programmers compared to Java programmers. The way they think. The things they have and haven’t read. They typically haven’t done unit testing, typically haven’t used any open source software. In general, their view of the software development world is quite shuttered. I know this is a gross generalization, but it’s noticeable. It’s a shame, because C# seems like a pretty cool language…) 

I quickly found Cuke4Nuke which allows you to implement your step definitions in C# rather than Ruby (the norm for Cucumber) and Cuke4VS which would give you syntax highlighting etc. when editing feature files in Visual Studio. Here I encountered my second couple of problems: Cuke4Nuke wasn’t ready for .NET 4 and Cuke4VS even to this day doesn’t work on VS2010. Of course our project was using .NET4 and VS2010. Fixing Cuke4Nuke wasn’t too hard, I fumbled my way through that OK. Fixing Cuke4VS though was beyond me. Well, beyond the amount of effort I wanted to put in anyway. I’ve not done much with C# and nothing with Visual Studio addins like this. I sorta got it working, but nothing I could push back as a quality patch to the Cuke4VS project.

Then I discovered Specflow. 

Specflow has some distinct goodness:

  • It’s Gherkin compatible, which means you describe your requirements in terms of features and corresponding given/when/then scenarios. While I lack the experience that would come from doing substantial real work in this, it intuitively feels “better” to me than the table based approach taken by Fit, Concordion etc.
  • It’s good to go with VS2010 and .NET4.
  • No Ruby required. With Cucumber, even though you can write your step definitions in C# courtesy of Cuke4Nuke, you still need a Ruby installation to parse your .feature files and call out over the wire to your C# steps. I’m not the first to observe that the .NET/C# community is kind of reluctant to install “weird shit” like Ruby that “doesn’t come from Microsoft”.
Sadly, at this point in time it also has some challenges:
  • Documentation. The main problem is the documentation, or lack thereof. Pull down the “Specflow guide” PDF and you’ll find a clear work in progress, incomplete and missing sections, editing notes etc.
  • Examples. Github hosts a SpecFlow-Examples project, but it needs some TLC. There are some reasonable examples in there, but quite where is not immediately obvious. All the action is inside ASP.NET-MVC/BookShop, not the features folder. Github is also potentially another barrier to entry for people. I don’t know how many C# developers are familiar with Github, but none on my team are.
  • VS silliness. Not Specflow’s fault, but I had hoped to use Visual C# Express for the QA people on my team to create .feature files. Sadly Microsoft has seen fit to not support Add-ins so it doesn’t work. Another barrier to adoption, at least for my team.
  • Stability. I never actually experienced this myself, but two of the engineers on my team said they had stability problems that they attributed directly to Specflow. It “mangled the project file.” Things “wouldn’t compile and it was complaining about Specflow stuff”. I never really got great details on this, so it’s hard to tell how much if any was directly down to Specflow. How much was me with rose-tinted glasses seeing no problem attributable to Specflow and perhaps others less interested in a new technique seeing every problem attributable to Specflow.
In general, you’re going to need a lot of patience, a genuine desire in a majority of the team to do something quite radically different to “normal” and you’ll need to draw a lot of information from the Cucumber community and then be comfortable using the mailing list for additional support. I’m actually OK with this, but it can be a real barrier to entry for a lot of people.

There were also some serious non-technical challenges; people skills and organization/political in nature.

Process and regulatory barriers. 
My team makes software that is subject to 21 CFR part 11 regulations. That’s stuff that tries to help make sure any pharmaceutical drugs you take are safe. That kind of industry cascades regulations through to any software related to clinical studies to make sure the software is of high quality. Our operation can be (and frequently is) audited by our clients to make sure we comply and can (though rarely is) audited by the FDA. Having devised a “winning formula” that gets us successfully through audits people are naturally reluctant to make changes in anything that would get seen by an auditor. Having your “requirements” in .feature files alongside source code then is a serious brown trouser moment for some people.
  • “They could end up seeing our source code!” Yes, believe it or not, a process designed to make sure the software involved in clinical trials doesn’t actually often have people look at the source code. They prefer Word documents, signed in duplicate (or more).
  • “We need to show them a traditional requirements specification with numbered requirements that we can trace through to test cases.” Because that’s the only possible way to do this. Of course.
  • “We have to review and sign off the test scripts before they’re executed, and prove that we have done so.” I can see how this made sense 20 years ago when test scripts where a documented set of steps a manual tester followed. But what does this even mean when your “test script” is a plain text .feature file and a corresponding set of step definitions in C#?
I actually have to commend the internal quality expert I spoke to for being very receptive to new ideas and trusting my assertion that I could probably generate a traditional style requirements specification and traceability matrix that mapped features to test cases from .features files. She was less resistant than some people on the team.

Skills Deficits 
The people on the team responsible for testing the product and generating the nice bundle of validation paperwork those who audit us like to see primarily have a background in testing software manually. I’d looked over their manual test scripts before (long Word documents with lots of boilerplate and then long rambling detailed steps with expected outcomes and places to mark discrepancies) and got the impression they could be better. They seemed to meander quite a lot, with a lack of clarity over exactly what precisely one was testing. I do now know how anyone could review them and spot test cases that were missing. I’d even taken a manual test script for a fairly trivial feature that ran to thirteen pages and felt pretty smug about being able to reduce it down to one short .feature file.

Some people on the team had asked me whether I thought those folks could use Specflow. Could they write good .feature files and the corresponding step definition classes? It was a fair question, and I was pretty sure some might struggle. That said, we already had a test automation tool called Test Partner that involved folks writing things in VBA, so if they could do that, could this be much harder? And I didn’t see this as a reason to indicate that Specflow and BDD was wrong. If BDD is a good idea it’s a good idea. Sure, maybe we needed to train folks or modify the composition of our team, but that was all.

But oh did I underestimate just how much they would struggle. At this point in time I wasn’t sure if Visual Studio Express would work with Specflow. So I suggested they just try. There then followed quite the flurry of emails. People who ostensibly test things for a living were unable to reliably determine whether they could use Visual Studio Express or a full featured version. 

And that was just the installation. 

Then we got into writing a few .feature files. The first one wasn’t…very good. This was a little disappointing as I’d circulated a number of links to articles on the web about writing “good” features but, OK, it was a first attempt. A couple of the software engineers very graciously (much more gracious than I’m being right now, but I’m currently enjoying my second Left Hand Brewing Company’s Fade to Black) pointed out some things they might do to improve them. Sadly subsequent attempts at writing .feature files were no better. It was as though the advice offered was ignored or not understood. 

We didn’t even get near them implementing a step definition. I really have to wonder what they’ll be putting together in Test Partner in those VBA scripts it uses. Thank goodness we have unit tests.

Incumbent Test Framework
As indicated above, we have available (actually we’re pretty much obliged to use) a product called Test Partner. Of course it’s only for QA people, especially given the per-seat cost. It’s all kinds of horrible so far as I can tell after looking at it. It doesn’t even make it easy to place your test scripts under source code control, preferring instead to keep them inside a database (but of course…) and though you can export them (base 64 encoded, naturally) it’s hardly ideal.

In addition to the original Test Partner investment, several people have spent a not inconsiderable amount of time and money building an additional layer (or framework) on top with functionality for such things as logging and managing collections (*cough* reinventing the wheel *cough*).

Of course the political reality attached to this state of affairs is that any other pretender to the crown has got its work cut out.

Cultural suitability
My love affair with computers started back in the early 80s around age 10. A friend had an Apple IIe computer and we played Bouncing Kamungas and other mad games, we entered epically long listings that sometimes worked. The school had a Commodore PET. I learned to program simple games in BASIC. I got a Vic 20 and wrote more games, even sold some to people at school. Then later I got an Amstrad CPC 6128 complete with it's wacky 3" disk drive and after that an Amiga. I worked in an independent computer store and wrote code for their business customers at age 16. I went to college and learned more. I got a “proper” job. It’s waxed and waned ever since. There’s still nothing quite like the fun of programming I think. But the reality of a corporate job can take the shine off that – it’s not the same as sitting there with your best buddy late at night debugging a game you’ve spent all day writing. So throughout my professional career my enthusiasm has varied along with different projects, languages, technologies, jobs and bosses. 

Nevertheless, several years back I took a leadership position, got interested in agile and discovered a new found passion for reading about and playing with new technologies and approaches to building software. I joined twitter, found a community of thought leaders and everyday practitioners, started reading books and blog posts on all of this, got excited, started a transition at work. Mostly successful. But not successful enough I guess. I’m still one of the very few people that does that kind of thing. And when you’re perhaps the only one on a team that reads and writes blogs, uses twitter, reads books, wants to try doing things differently it can be tough. You're seeing so much that other people aren't.

The fact is I’ve been having to sell BDD to almost everyone. Maybe one person really “gets” the potential. A few more are fairly excited about it. But it's so far outside their comfort zone.

When I first managed to get some consensus that we should try this I hadn’t done enough with Specflow myself, almost all my experimentation up to this point was with Cucumber. I wasn’t worried though, the guy that was going to try it was an excellent engineer. I was more than a little surprised then when almost every hour a new “issue” was presented on why this couldn’t possibly work. I hadn’t seen that coming. 

Culturally, my current team just isn’t ready or interested in something like this. And I have to accept that I can’t necessarily change that. Maybe I should have done a lot more work with Specflow myself first and had good examples to show people. I'm not sure.

Blah, blah whine! WTF am I gonna do about it?
I don’t know yet. Not for sure. I’m thinking maybe I can contribute to the Specflow project and help out with the documentation. This isn’t entirely appealing though if we won’t even be using it. So maybe I’ll use it on the sly, even if we’re not “officially” using it for our project maybe I can keep trying it out. And use that experience to develop something like the documentation that people on my team felt needed to be there. 

If you have any suggestions, let me know...

OK, self-indulgent rant over.

Published at DZone with permission of Jon Archer, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Andy Doddington replied on Wed, 2013/01/30 - 9:34am

"I do now know how anyone could review them and spot test cases that were missing"

now vs. not - ironic or what? ;-)

Ken Cline replied on Wed, 2013/01/30 - 10:33am in response to: Andy Doddington

No, Andy ... I think he really meant "I do not know how...". If you've ever seen the test cases he's describing, you'd understand. They can run on for many pages and go into excruciating detail on exactly what you're supposed to do and the expected result. The problem is, they're so complex and verbose that it's nearly impossible to tell if you have full coverage of even the one thing you're testing - much less everything that needs to be tested. 

They're also impossible to maintain. In most cases, they're married to the UI. If something gets changed (the text of a message, the location of a control, etc.), the test case needs to be updated to reflect the change. I've lived that nightmare & don't want to go back!

Eugene Dvorkin replied on Wed, 2013/01/30 - 9:43pm

Hi Jon,

Very well done post. It's hard to bring change into enterprise. It's takes a lot of time, patience and persistence. But I notice one thing - if your people are not completely dump, they will learn, they understood that they will be replaced by offshore test monkeys or automation sooner than later.  They have to embrace automation (BDD or not)  and learn how to do it now. My company had a layoffs, a lot of QA people got fired but amount of work did not decrease. Now, who left, are much more acceptive to idea of BDD and test automation. You know, some people need stress to move forward. My advice - keep doing it, keep introducing new ideas. Gradually, patiently. People will get influenced by you ideas. I wish you good lack, you post make me feel better.



Andy Doddington replied on Thu, 2013/01/31 - 2:43am in response to: Ken Cline

Hi Ken,

Yes, I realise that - that's why I highlighted the word not and suggested that the word now was intended instead. Maybe I didn't make myself clear enough.

I also understood the context, which is why I suggested that the error was ironic, appearing as it did in the midst of a large-ish body of text, where it could easily be missed. I too have lived the nightmare...

Senthil Balakrishnan replied on Fri, 2013/02/01 - 6:51pm

One of the key challenges of software development is to tie a requirement to code & to an acceptance test. I felt BDD does an excellent job to bridge this gap. I have used cucumber/BDD as part of an agile development on a java based web project. I do understand after a point the scenarios & step files can go unmanageable, but then wat are the options ?

Alan Parkinson replied on Sun, 2013/02/10 - 2:55pm

Thanks for the interesting article Jon.

Traceability and Regulation compliance for BDD/Gherkin based tools is something we are working on adding to Behave for JIRA. If you have any other requirements I would be happy to discuss them.

Andrian Bulat replied on Wed, 2013/02/20 - 10:42am in response to: Alan Parkinson

After doing BDD for almost 2 years using SpecFlow we have gained some experience with it.

We have a slightly different approach: 

QA + DEV+ BA, all together they are responsible for defining the Scenarios + all test cases inside the scenarios, all Given When Then stuff

And automation, the hardest part of BDD, is done by developers not by QA, using all necessary components, we ended up using web services and WatIn for UI testing.

In the beginning it's hard for everyone to start doing it, but in the end brings more value than UnitTesting.

Bernie Woolfrey replied on Tue, 2013/06/18 - 6:27pm

 Thanks for the post, and the comments. This is very timely as I'm trying to introduce BDD and Cucumber/Gherkin into our world. We are gradually getting buy-in. First with devs, then testers. Our next challenge is the BAs.

Henkz Sall replied on Wed, 2013/06/19 - 8:33pm in response to: Bernie Woolfrey

Steve Jobs autobiography a "must read" for web designers and developers?
Will this ridiculous adoration never end? Other than that complete and utter "odd one out", the rest is a good list of recommendations is bbom

Henkz Sall replied on Sat, 2013/08/03 - 9:53pm

The tea master bowed, and politely step into the gutter to let the fearsome ones pass. But although one warrior went by, the other remained rooted to the spot. He stroked a long black whisker that decorated his face, gnarled by the sun, and scarred by the sword. His eyes pierced through the tea maker’s heart like an arrow. Cod here acompanhantes em sp

Carolina Alves replied on Thu, 2013/08/29 - 12:24pm

Thanks for posting such a nice article. I love the way you have described the whole article. Thanks and keep posting new things.

Carolina Alves replied on Fri, 2013/08/30 - 9:51am

Very efficiently written information. It will be beneficial to anybody who utilizes it, including me.

Carolina Alves replied on Tue, 2013/09/03 - 9:18am

Just loved all of the tips you have so lovingly spelled out for us (your on line family). A big Thanks for that. I would definitely take these in mind.

Henkz Sall replied on Tue, 2013/09/03 - 8:11pm

That would be actually an interesting research topic, how digital has changed the way people read and write, i thing Camiseta Hollister

Carolina Alves replied on Thu, 2013/09/05 - 10:00am

The tea master bowed, and politely step into the gutter to let the fearsome ones pass. But although one warrior went by, the other remained rooted to the spot.

Vera Loiola replied on Thu, 2013/09/05 - 6:17pm

This is something really interesiting. I was looking for something different but seo related and I found this article which of course is worth to try.
Thanks for sharing

Loris Smith replied on Thu, 2013/09/26 - 2:37am


V on Shenton
Very efficiently written information. It will be beneficial to anybody who utilizes it, including me.

Mari Del replied on Tue, 2013/10/08 - 9:08am

Just loved all of the tips you have so lovingly spelled out for us (your on line family). A big Thanks for that. I would definitely take these in mind.


Mari Del replied on Tue, 2013/10/15 - 9:36am

I would like to say that this blog really convinced me to do it ! Thanks, very good. We are really happy just for this post in this website.

Henkz Sall replied on Thu, 2013/11/14 - 8:39pm

Oh right, me. Loving me for living Apartamento aguas claras

Amar Najah replied on Fri, 2013/11/29 - 3:44pm in response to: Andy Doddington

OK , i am agree with you

Mari Del replied on Fri, 2014/03/21 - 3:05pm

I did enjoy reading articles posted on this site. They are impressive and has a lot of useful information.

Carlos Ferreira replied on Sat, 2014/04/26 - 12:45am in response to: Ken Cline

Thanks for post. 

Great site.

Filmes Online  Assistir naruto Assistir futebol  Filmes online 

Neozinho Guerreiro replied on Sat, 2014/04/26 - 2:18am in response to: Andy Doddington

nice comment, thank you  Assistir Filmes Online 

Neozinho Guerreiro replied on Sat, 2014/04/26 - 2:20am in response to: Andy Doddington

lol, yes, nice man Ver Filmes Online 

Neozinho Guerreiro replied on Sat, 2014/04/26 - 2:22am in response to: Andy Doddington

yes, coment interessante, valeu mesmo Ver Futebol Ao Vivo 

Mari Del replied on Wed, 2014/05/14 - 2:27pm in response to: Eugene Dvorkin

This is my first opportunity to visit this website.Thanks for sharing useful information .

Commenter Man replied on Fri, 2014/11/21 - 2:12am in response to: Ken Cline

consumers by a consumer, this fan site includes a fun and informative Diabetes Miracle review, ebook FAQ, and PDF download guide that provides an insider's look at the program and answers frequently asked questions about Dr. Robert Evans and Paul Carlyle's unique and highly popular diabetes treatment guide droit constitutionnel

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.