The Apache Software Foundation Blog

Monday March 05, 2018

Success at Apache: Open Innovation from a Non-native English Country

by Von Gosling

When I saw the "Success at Apache" series, I thought about writing something about my, being from a non-native English country, Open Source experience these past few years. Last year, RocketMQ graduated from the Apache Incubator and became one of the Apache Top-Level Projects. As one of the original co-founders of RocketMQ, I was proud to see an Open Source community from Apache RocketMQ that has an ever-growing diversity. The Apache Software Foundation (ASF), one of the most famous and great technology brands, has thousands of companies’ software infrastructure based on their projects. This is proven from the worldwide download mirror activity in ASF statistics. As an early implementer/pioneer of Open Source in China, Apache HTTP Server, Apache Tomcat, Apache Struts 1.x, and Apache Maven are my favorite software stacks when I worked for building distributed and high-performance websites.

Last year, I wrote an article about the road to the Apache TLP, which is published in China’s InfoQ. Some people asked me how to be more ‘Apache’ and how to build a more diverse community. These are the questions that many people are concerned about. In this blog post, I will address how to be more collaborative around the world, especially in non-native English countries.

Open Communication

With more and more instant messaging apps coming up in Android and IOS world, the younger generation prefers to communicate using such way, which has spread to the daily coding life for the majority of people. But, it is not search engine friendly and in most cases it does not support multi-channel for multi-language. I have been involved in many such local technology groups, together we have discussed what went wrong, explored ideas about how to solve it, and come up with a good solution together. This method worked for all my past projects, but when we hope to be more involved in Open Source around the world, that method does not work well. I remember clearly when RocketMQ began to discuss the process for its proposal, some people complained about what we have to do in the local community. We learned much about from this discussion in the community, and thus, found an effective solution. Hence in the Apache RocketMQ community, we encourage users to ask the question using the user email list. In order to make the communication process effective, we answer the question in the same language of the question. With more and more committers coming from different countries, this solution will help to grow the more diverse community. But, as John Ament said in another "Success at Apache" post https://s.apache.org/x9Be --open communication isn't for everything. We also allow private communication between the users and us as some questions might not be proper to discuss publicly. But that isn't a part of the decision making process. Likewise, anytime we're talking about individuals in either a positive or negative way should be conducted on the private list for a project.

Easy ways to be involved in the community

This is another top concern in the Open Source world. Some people may not know that in China there are many local communities about Apache Projects, such as Apache HTTP Server, Apache Tomcat, Apache Spark, and Apache Hadoop. Such Projects have corresponding Chinese documentations. On the other hand, we try our best to improve the English documents. We consider the messages behind every document page. If one finds a minor or big native narrative polish, one could leave a message, or send feedback to our dev or user email list. Besides documentation, we also hold programming marathons in the community irregularly to get more involved with the community. We could find more users who have more interest, especially cross-domain technology in such campaigns. Recently, we open sourced more tasks in the Google Summer of Code. Students will develop Open Source software full-time for three months. We will provide mentoring and project ideas, and in return have the chance to get new code developed and --most importantly-- to identify and bring in new committers. It is another chance to let PMC members know how to improve and let more students get involved in the community easily.

In China, Internet giants like Alibaba are devoting themselves into Open Source projects hence according to my personal experience, it made sense to help more excellent Chinese projects to come into the Incubator. Right before the Lunar New Year, another famous project from China, Dubbo, started its Apache journey. I am glad to be a local mentor and hope to continue to share what we have learned. Thanks to the ASF, more and more Open Source projects will benefit our daily coding. That is a great appeal around the world’s Open Source field.

Von Gosling is a senior technology manager working at Alibaba Group. He has extensive industry software development experience, especially in distributed tech., reliable Web architecture and performance tuning. He holds many patents in the distributed system, recommendation etc. he has been a frequent speaker at Open Source and architect conferences worldwide including ApacheCon and QCon. He has been the lead for messaging at Alibaba as well as the Tenth and Sixteenth CJK OSS Award recipient. He is the original Apache RocketMQ co-founder and Linux OpenMessaging Standard Initiator.

= = =

"Success at Apache" is a monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be 8) Meritocracy. https://s.apache.org/DiEo 9) Lowering Barriers to Open Innovation https://s.apache.org/dAlg 10) Scratch your own itch. https://s.apache.org/Apah 11) What a Long Strange (and Great) Trip It's Been https://s.apache.org/gVuN 12) A Newbie's Narrative https://s.apache.org/A72H 13) Contributing to Open Source even with a high-pressure job https://s.apache.org/lM9O 14) Open Innovation from a Non-native English Country https://s.apache.org/lh61

# # #

Posted at 03:00PM Mar 05, 2018 by Sally in SuccessAtApache | |

Monday February 26, 2018

Success at Apache: Contributing to Open Source even with a high-pressure job

by Anthony Shaw

I believe in the mission of the ASF for many reasons, but the first is the reason why I got into open-source software- free and open access to knowledge.

Back when I was age 12 (1998), I started to learn to program in dBase 4. dBase 4 and the compiler Clipper were not cheap, especially for a $5-a-week paper-round. The box with the software was unwanted by a local company and it came with the manuals. We didn't have the internet at home yet so I was left to go by the manual, and what I could find from second-hand stores and office cleanout sales. For the next decade, I learnt to case based on what I could find, borrow and scavange until in 2002 when I got a copy of Linux and assembled a couple of machines from unwanted parts from the village computer store.

This is where I discovered free and open-source software and really started to build on my coding skills.

My goals were to learn and to share what I'd learnt that others could get to where they needed to go faster. It also helped that software skills were well sought-after in Europe so it set off me in a career in IT.

20 years after I learnt to code, I've moved out of software-engineering and into Learning and Development at Dimension Data for a 29,000 person technology company that operates in 49 countries across the world. My current roles involves about 3-months a year of travel (15 countries typically), managing a department of over 30 people spread across 4 countries and 4 timezones and delivering on large and complex initiatives with high-degrees of change and short deadlines.

In 2016 I made a choice after getting promoted into my current role that I would continue to contribute to the open-source projects I'd worked on for years. But I set myself 3 rules;

1. I would not take away from time with my family

2. I would not interfere with my work commitments

3. I would look after my health

My open-source contributions

For the past 4 years I've made around 1,000-2,000 contributions annually. These have consisted of bug fixes, submissions, and to around 50 projects.

The largest contributions I've made have been to Apache Libcloud, a multi-cloud abstraction library written in Python. Initially this was driven by a work commitment to contribute an integration with the cloud API we'd designed, but I soon realised the power of the library. Going back to my original goal of free and open access to knowledge, I'd seen an alarming trend in the computing world. Proprietary APIs were driving what is known in the industry as "stickiness" or to be frank, lock-in.

Cloud lock-in means that anyone without access to a reliable network, money or willing to sign up to these contracts is being pushed out of advances in technology. I know developers that are students, in remote areas such as rural Australia, Asia and Africa, or those who simply have little money.

Apache Libcloud's design means that you can design applications which can be deployed to OSS platforms like Apache CloudStack and OpenStack.

After finishing the work driver around 100 hours developing a container abstraction layer for Apache Libcloud that meant that developers could write automation for OSS platforms like Kubernetes using the same API as you would with a public cloud provider.

This was all whilst managing family time, work commitment and my health.

These are my 3 tips for maintaining contributions with a high-pressure job:

1. Pick a project that you care about

This is the most important, something that just sparks your curiosity is good fun, but long term interest often dwindles. I've been victim of "ooh shiny thing" many times in the past, but as my career has taken off, I've had to develop the discipline to stop myself from writing my own scripting language, or building an automated sprinkler system from scratch. I stop and remind myself that I might have the time this second, but what about next week and next month? Stop and prioritise.

Prioritise projects that mean something to you.

The 2 OSS projects I commit the most to are Apache Libcloud and SaltStack. I believe in Apache Libcloud's mission of giving open-access to cloud platforms. My SaltStack contributions have been focused around cloud abstraction, networking API abstraction and other fixes and utilities that make it easier for developers and end-users.

The difference between picking something shiny and something you believe in is that long-term you commit more and you find it easier to jump in and help when you can. But how do you find the time?

2. Choosing your tasks wisely and making time

I get asked this question all the time, "how do you find the time". When I try and convince people to contribute to OSS the response is always about time.

Get rid of the things that don't add value

If you can afford to, hire help to give you back time in your week. Not only does open-source help with your skills and knowledge, but it increases your value to a potential employer. Hiring someone to blow the leaves, or help with the chores once a week doesn't need to cost a lot, but if you work out how much value you can get back from that time it often makes sense.

Another thing I've been strict about is binge-watching TV series and gaming. Playing 100-hours of the latest game might be fun, but I find developing more rewarding in the medium-to-long term. Find ways to unwind that don't consume so much time, like meditation, exercise, or reading.

But, if you do need to put your feet up and watch some TV for a few hours, don't feel guilty about it.

Work smart, not hard

When I do sit down to contribute something, it'll have been carefully planned and thought through what I'm going to do, what I'm going to test and how I'm going to structure it. I try and complete tasks quickly, with foresight and a goal. Once I've completed this 1 module, with tests, I'll submit my contribution. Don't try and refactor the whole project over a weekend. Keep it simple.

But we all know sometimes the best plans go out the window. If you find yourself going down one of those rabbit holes, where you can't get something to compile or you can't debug one of those zombie bugs we love so much as developers.

Stop yourself.

You can easily sit until 3am banging your head against the wall trying to figure it out. This was my advice when I used to manage development teams. If you get stuck, take a break, ask for help and if that still doesn't work, move onto something else.

Sometimes I pause working on a task if I can't figure it out. Pause for an hour, a week, or even a whole year. When you have one of those "aha" moments, you go back in and finish the job.

It saves time, it delivers better software and it's a good skill to have as a developer.

Find time

A contribution comes down to 3 things:

1. An idea

2. An understanding

3. A "change", like a fix, feature, test, code-review, documentation etc.

The ideas come to me through reading, listening to users or looking at bug submissions. I do this as and when I have a spare minute. This is normally on my lunch break, when I'm waiting for someone or something.

The time for understanding I get by listening to podcasts and talking to people at conferences. I get a few hours a week in the car and I spend time doing some chores. During that time I always have headphones on to listen the newest Python podcast or OSS update.

The time to sit down and write, code, or test comes for me on the plane (where I'm writing this blog post!). Last year I did enough miles in the air to fly around the world 8 times, most of that time was spent coding, relaxing or sleeping. Aside from that, time spent in airport lounges, on the train or waiting for people I'll whip out my laptop. Any plane that has Wi-Fi I can push changes, else the minute we land I'll have a laptop open and running git push.

Weekend-time is off limits unless I'm travelling or I'm alone. That's rule 1 -- do not take away from time with the family.

3. Managing your workload and avoiding burnout

There are 2 components to this, managing your work commitment and managing your contributions. You need to do both to succeed.

It's ok to stop and take a break. There is always a pull-request to merge, a bug to inspect, and an email from an end-user. If you need to take a break for a while, talk to the team, ask for help and be frank. We're all in the same boat, contribution is optional.

So many times I see people contribution feeling like they have a complete obligation to test and fix bugs at 2am

and then go to work at 8am. This is normally because they care about the project, they care about quality and they care about their reputation but sometimes you need to step back.

A strong project community will step up and help. If you know that work is going to be tough for the next few months, tell the team and set yourself a limit. Wind back for a bit until things calm down.

Managing work commitments is tough, because there are often financial consequences (or at least a perception of them).

After 7 hours, you're not really adding value. I used to have a lounge-chair next to my desk and now I have a hammock as I work from home. After a few hours of solid concentration I'll happily go and sit down and do nothing for an hour. Your brain needs a break, sure you'll get the odd "working hard" jab from a passer by but I'm working smarter not harder. Once I'm refreshed I'll finish the next task about 30-40% quicker, to a better level of quality and insight. On the occasion I've done 12-14 hour work days, my brain is shutting down to conserve energy and your critical thinking is the first thing to switch off. Followed by logical thinking, this is where you make mistakes and deliver work that is less than a quality you'd normally expect.

I live close to the beach so my time out is going for a swim in the ocean or spending a bit of time with my family. As a manager I also see a responsibility to make it clear that it's encouraged to step back and recharge. Just in our chat-channel to say that I'll be offline for a couple of hours as I'm going to the beach mid-afternoon. I don't feel guilty about it and I hope they do the same.

Learn how to say no and don't feel guilty about it. When I coach people on this I ask, "who asked you to do this? Was no an option? What value is there in delivering this? What is consequence of not doing it? Who else could do it?"

Everyone wants to be helpful and indispensible, but your reliability is just as important to your reputation and what you deliver.

Conclusion

Look after your health, be smart with your time and contribute for a cause.

Anthony Shaw is the Group Director of Innovation and Talent Development at Dimension Data, an NTT company. Anthony is an open-source advocate, member of the Apache Software Foundation and Python Software Foundation and active contributor to over 20 open-source projects including Apache Libcloud and SaltStack. At Dimension Data, Anthony is driving digital transformation for Dimension Data’s global clients across 50 countries and 30,000 employees. Key initiatives are software skills, automation, DevOps and Cloud. Anthony is based in Sydney, Australia and blogs about skills, software and automation to 170,000 readers annually.

= = =

# # #

Posted at 11:02AM Feb 26, 2018 by Sally in SuccessAtApache | |

Monday February 05, 2018

Success at Apache: A Newbie’s Narrative

by Kuhu Shukla

As I sit at my desk on a rather frosty morning with my coffee, looking up new JIRAs from the previous day in the Apache Tez project, I feel rather pleased. The latest community release vote is complete, the bug fixes that we so badly needed are in and the new release that we tested out internally on our many thousand strong cluster is looking good. Today I am looking at a new stack trace from a different Apache project process and it is hard to miss how much of the exceptional code I get to look at every day comes from people all around the globe. A contributor leaves a JIRA comment before he goes on to pick up his kid from soccer practice while someone else wakes up to find that her effort on a bug fix for the past two months has finally come to fruition through a binding +1.

Yahoo – which joined AOL, HuffPost, Tumblr, Engadget, and many more brands to form the Verizon subsidiary Oath last year – has been at the frontier of open source adoption and contribution since before I was in high school. So while I have no historical trajectories to share, I do have a story on how I found myself in an epic journey of migrating all of Yahoo jobs from Apache MapReduce to Apache Tez, a then new DAG based execution engine.

Oath grid infrastructure is through and through driven by Apache technologies be it storage through HDFS, resource management through YARN, job execution frameworks with Tez and user interface engines such as Hive, Hue, Pig, Sqoop, Spark, Storm. Our grid solution is specifically tailored to Oath's business-critical data pipeline needs using the polymorphic technologies hosted, developed and maintained by the Apache community.

On the third day of my job at Yahoo in 2015, I received a YouTube link on An Introduction to Apache Tez. I watched it carefully trying to keep up with all the questions I had and recognized a few names from my academic readings of Yarn ACM papers. I continued to ramp up on YARN and HDFS, the foundational Apache technologies Oath heavily contributes to even today. For the first few weeks I spent time picking out my favorite (necessary) mailing lists to subscribe to and getting started on setting up on a pseudo-distributed Hadoop cluster. I continued to find my footing with newbie contributions and being ever more careful with whitespaces in my patches. One thing was clear – Tez was the next big thing for us. By the time I could truly call myself a contributor in the Hadoop community nearly 80-90% of the Yahoo jobs were now running with Tez. But just like hiking up the Grand Canyon, the last 20% is where all the pain was. Being a part of the solution to this challenge was a happy prospect and thankfully contributing to Tez became a goal in my next quarter.

The next sprint planning meeting ended with me getting my first major Tez assignment – progress reporting. The progress reporting in Tez was non-existent – "Just needs an API fix," I thought. Like almost all bugs in this ecosystem, it was not easy. How do you define progress? How is it different for different kinds of outputs in a graph? The questions were many.

I, however, did not have to go far to get answers. The Tez community actively came to a newbie's rescue, finding answers and posing important questions. I started attending the bi-weekly Tez community sync up calls and asking existing contributors and committers for course correction. Suddenly the team was much bigger, the goals much more chiseled. This was new to anyone like me who came from the networking industry, where the most open part of the code are the RFCs and the implementation details are often hidden. These meetings served as a clean room for our coding ideas and experiments. Ideas were shared, to the extent of which data structure we should pick and what a future user of Tez would take from it. In between the usual status updates and extensive knowledge transfers were made.

Oath uses Apache Pig and Apache Hive extensively and most of the urgent requirements and requests came from Pig and Hive developers and users. Each issue led to a community JIRA and as we started running Tez at Oath scale, new feature ideas and bugs around performance and resource utilization materialized. Every year most of the Hadoop team at Oath travels to the Hadoop Summit where we meet our cohorts from the Apache community and we stand for hours discussing the state of the art and what is next for the project. One such discussion set the course for the next year and a half for me.

We needed an innovative way to shuffle data. Frameworks like MapReduce and Tez have a shuffle phase in their processing life cycle wherein the data from upstream producers is made available to downstream consumers. Even though Apache Tez was designed with a feature set corresponding to optimization requirements in Pig and Hive, the Shuffle Handler Service was retrofitted from MapReduce at the time of the project's inception. With several thousands of jobs on our clusters leveraging these features in Tez, the Shuffle Handler Service became a clear performance bottleneck. So as we stood talking about our experience with Tez with our friends from the community, we decided to implement a new Shuffle Handler for Tez. All the conversation points were tracked now through an umbrella JIRA TEZ-3334 and the to-do list was long. I picked a few JIRAs and as I started reading through I realized, this is all new code I get to contribute to and review. There might be a better way to put this, but to be honest it was just a lot of fun! All the white boards were full, the team took walks post lunch and discussed how to go about defining the API. Countless hours were spent debugging hangs while fetching data and looking at stack traces and Wireshark captures from our test runs. Six months in and we had the feature on our sandbox clusters. There were moments ranging from sheer frustration to absolute exhilaration with high fives as we continued to address review comments and fixing big and small issues with this evolving feature.

As much as owning your code is valued everywhere in the software community, I would never go on to say “I did this!” In fact, “we did!” It is this strong sense of shared ownership and fluid team structure that makes the open source experience at Apache truly rewarding. This is just one example. A lot of the work that was done in Tez was leveraged by the Hive and Pig community and cross Apache product community interaction made the work ever more interesting and challenging. Triaging and fixing issues with the Tez rollout led us to hit a 100% migration score last year and we also rolled the Tez Shuffle Handler Service out to our research clusters. As of last year we have run around 100 million Tez DAGs with a total of 50 billion tasks over almost 38,000 nodes.

In 2018 as I move on to explore Hadoop 3.0 as our future release, I hope that if someone outside the Apache community is reading this, it will inspire and intrigue them to contribute to a project of their choice. As an astronomy aficionado, going from a newbie Apache contributor to a newbie Apache committer was very much like looking through my telescope － it has endless possibilities and challenges you to be your best.

Kuhu Shukla is a software engineer at Oath and did her Masters in Computer Science at North Carolina State University. She works on the Big Data Platforms team on Apache Tez, YARN and HDFS with a lot of talented Apache PMCs and Committers in Champaign, Illinois. A recent Apache Tez Committer herself she continues to contribute to YARN and HDFS and spoke at the 2017 Dataworks Hadoop Summit on "Tez Shuffle Handler : Shuffling At Scale With Apache Hadoop". Prior to that she worked on Juniper Networks' router and switch configuration APIs. She likes to participate in open source conferences and women in tech events. In her spare time she loves singing Indian classical and jazz, laughing, whale watching, hiking and peering through her Dobsonian telescope.

= = =

# # #

Posted at 11:00AM Feb 05, 2018 by Sally in SuccessAtApache | |

Tuesday December 12, 2017

Success at Apache: What a Long Strange (and Great) Trip It's Been

By Jim Jagielski

It is normally during this time of year that people get awful retrospective. We look over the last 12 months and come to terms with what kind of year it has been. We congratulate ourselves on the good and (hopefully) learn from the bad. We basically assess the ending year and start planning, even a little bit, on the one to come.

In general, we reminisce.

I am thinking not about 2017, however, but instead of 1995 and the origins of The Apache Software Foundation. And what a long, strange, and great trip it's been. And how incredibly lucky I've been to be a part of it.

A common saying is that success is mostly about being there at the right place at the right time, and although I'm not sure about the "success" part, it certainly applies to me. At the time I was working at NASA and was starting off a side business as an ISP and Web Hoster, and using the old NCSA web-server. I had created a small reputation for myself as an "expert" on a flavor of UNIX called A/UX, which was Apple's UNIX offering at the time. In addition to being the editor of the FAQ for A/UX, I also ported a bunch of "free software" to that platform and that's how I got started with Apache, providing patches to support A/UX, which is what I used as my web hosting platform. It was really no different than what I did for other software projects at the time.

And then something wonderful happened. I got hooked.

I really, really enjoyed the people I was collaborating with. I wasn't an "outsider" providing patches, I was part of the inner circle. I was a full fledged member of the Apache Group. I started to really understand just how all this really could change the world, and how I could maybe be a small part of it.

As a result, Apache changed my life, literally. Instead of doing software development as a way of "getting my job done" (at NASA, I was a power system engineer, and so I would code modeling and simulation software for spacecraft solar arrays, batteries and orbital mechanics), I starting doing software development as my job, in addition to my hobby. Apache and Open Source became a huge part of my life, and my career changed to focus on Open Source almost primarily, a change that continues to this day.

During this time I've been fortunate enough to work with, and learn from, extremely talented people. Not only related to code, but legal matters, inter-personal skills, presentation skills, etc. I've had opportunities that I never imagined and met people I never would have had expected otherwise. I'm made great friends. I've been mentored by incredibly giving people and have mentored in return. And have seen my mentees become mentors themselves.

Over the years, I've seen Apache grow from a rag-tagged group of people working on a web server to one of the leading Open Source foundations in the world with more than 300 projects under our belt. I've been blessed to serve on the board of the ASF for every single year since we incorporated in 1999, seeing 2nd and now 3rd "generation" Apache Members take on the reins.

The Open Source movement, and especially Apache, have given more to me than I could ever pay back, and that is why I still volunteer and contribute. Of course, to be honest, I still get a kick out of it, and love what I am doing, and continue to enjoy the opportunities and, especially, the people that I get to work with.

But, you see, I'm nothing special. All this is also open and available to you. You too can change the world, and have your world changed in return. We all have talents that can be shared, talents that can be recognized and rewarded. Apache is a family, always looking for new family members.

So take that first step. Find a project and community you want to a part of. Jump in. Have fun. Grow. Learn. Teach. Live.

But just be prepared to get hooked, and have your life change.

Jim Jagielski is a well known and acknowledged expert and visionary in Open Source, an accomplished coder, and frequent engaging presenter on all things Open, Web and Cloud related. As a developer, he’s made substantial code contributions to just about every core technology behind the Internet and Web and in 2012 was awarded the O’Reilly Open Source Award and in 2015 received the Innovation Luminary Award from the EU. He is likely best known as one of the developers and co-founders of the Apache Software Foundation, where he has previously served as both Chairman and President and where he’s been on the Board Of Directors since day one. Currently he is Vice-Chairman. He's served as President of the Outercurve Foundation and was also a director of the Open Source Initiative (OSI). Up until recently, he worked at Capital One as a Sr. Director in the Tech Fellows program. He credits his wife Eileen in keeping him sane.

= = =

# # #

Posted at 12:10PM Dec 12, 2017 by Sally in SuccessAtApache | |

Wednesday October 25, 2017

Success at Apache: Scratch Your Own Itch.

By Ignasi Barrera

Recently I was at an industry conference and was happy to see many people stopping by the Apache booth. I was pleased that they were familiar with the Apache brand, yet puzzled to learn that so many were unfamiliar with The Apache Software Foundation (ASF).

It's important to recognize not just Apache's diverse projects and communities, but also the entity behind their success.

Gone are the days when software, and technology in general, was developed privately for the benefit of the few. As technology evolves, the challenges we face become more complex, and the only way to effectively move forward to create the technology of the future is to collaborate and work together. Open Source is a perfect framework for that, and organizations like the ASF carry out a decisive role in protecting its spirit and principles.

The ASF's mission is to provide software for the public good. We take it one step further, by giving all our Open Source software away for free. According to this mission, the foundation was established back in 1999 as a US 501(c)(3) non-profit charitable organization, and constitutes an independent legal entity to which companies and individuals can donate resources and be assured that those resources will be used for the public benefit. Its all-volunteer nature, along with the meritocracy model followed by its communities, are the pillars of the neutral, trusted space where Apache software is developed.

We strongly believe that good software is built by strong communities. Successful Open Source projects are the result of the work and collaboration in their communities and the people behind them. It is all about the people. Experience has shown us that helping people work together as peers is key in producing software in a sustainable way, and we have collected the lessons learned all these years in what we call "The Apache Way".

This Apache Way is a set of core behaviors all Apache projects follow that are designed to ensure projects are independent and diverse, and that anyone can participate no matter what gender, culture, time zone, employer, or even expertise they have. One can start collaborating with a project by contributing patches or implementing new features, but merit is not only measured by code contributions. Helping users, improving documentation, promoting the project, and other non-coding activities are very valuable and recognized as such, and the recognition of this merit and implication is expressed by granting more privileges in the project: from commit access, to invitations to join the Project Management Committee, to invitations to join the ASF Membership. One of the great differentiators between the ASF and other open source foundations is that the ASF does not dictate the technical direction of its projects: each Apache project is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides their respective project's day-to-day operations, including community development and product releases. Meritocracy drives the growth of the communities, and ensures anyone can contribute to projects that are ruled by the people who is involved and really cares about them.

Learning to work this way is not always easy, though. Projects come to the Foundation from very different backgrounds and whilst some of them already have communities that are used to collaborate in open ways, others find it challenging to embrace these core behaviors. The Apache Incubator is the main entry point for codebases and their communities wishing to officially become part of the Foundation, and is where they learn how to put all these principles in practice. Some will find this way of working a good way to rule a project and will graduate as an Apache top-level project, some may find that the Foundation is not the best option for them and choose to leave. Both options are good outcomes, as projects will have invested time in thinking about their community model and how they want governance to be, and this always benefits the Open Source world.

This Open Source model not only exists to create sustainable Open Source projects, but also to meet the expectations of the rest of the world. Software developed at Apache comes with a set of guarantees granted by the popular and business-friendly Apache License, but also with others that are the product of this open governance model, such as project independence or a well-defined project lifecycle. The ASF not only defines how projects operate while active, but also what happens when a project reaches its end-of-life, which is also important for adoption but often not considered by Open Source projects.

These guarantees, along with the reputation earned by many years of producing high-quality open source software, make the +300 freely available Apache projects, from Abdera to HTTP Server to Hadoop to Zookeeper, a trusted choice for individuals and companies looking for Open Source solutions.

The saying "Scratch Your Own Itch" is popular in the tech space, and is an integral principle at the ASF. Apache Committers have a responsibility to the community to help create a product that will outlive the interest of any particular volunteer, as well as for helping to grow and maintain the health of the Apache community.

As an ASF Member, I'm helping with project outreach and mentoring new individuals that make up the greater Apache community.

The Apache Software Foundation provides a safe place for Open Source development, and will keep evolving as technology evolves, welcoming all kinds of projects and communities, and helping people embrace Open Source. Let's see what the future holds for the Open Source world and how we can contribute to making it a better place. Scratch your own itch.

Ignasi Barrera is a long-term Open Source contributor and became involved with the ASF in 2013, when jclouds was first submitted to the Apache Incubator. He is a member of the Apache jclouds Project Management Committee and still actively contributes to the project. Ignasi became an ASF Member in 2015, and helps with community development activities and the promotion of Open Source.

= = =

# # #

Posted at 10:06PM Oct 25, 2017 by Sally in SuccessAtApache | |

Monday October 02, 2017

Success at Apache: All My Roads Led to Apache

by Pat Ferrel

I became involved with Apache in 2011. After several years in startups where, as CTO, I felt too removed from building things. Looking for a change, I was keenly aware that the most interesting thing about the startups was our early use of Machine Learning techniques and I wanted to see if building ML solutions, for companies new to the field might not be more satisfying. I started by spending nearly a year in researching the type of applications we had needed in the startups: Natural Language Processing (NLP), text analysis, clustering, and classification. In those days Apache Mahout http://mahout.apache.org/ had several good solutions that were designed for Big Data and approachable by an individual. These ideas seem fairly commonplace now but were in early days only 6 years ago.

Given a great platform to experiment with, I built a web site to advertise expertise in ML but also to showcase many examples from my experiments, including a topic-oriented content site based on clustered and classified text that used NLP to add entities to text. I blogged about things I had learned and techniques that produce results.

Then I got the first contact about a project and it was from a completely unexpected direction: recommenders. Fortunately Apache Mahout then had the state-of-the-art OSS suite of recommenders so I took the consulting job. The company had rolled their own recommender and was selling it as a service but it was old and they wanted to investigate replacing it.

Welcome to Big Data

The nature of recommenders means you deal with huge amounts of data because you have to track several million people’s actions over years. We had data from a large online retailer and were tasked with using this data to beat the in-house recommender. Specifically they wanted to see if they could improve performance (better results and faster compute times) and get something easier to maintain.

The first job of a good consultant is to define the problem and outline a path to resolution that fits with the company’s competencies. To me this meant looking at the current system and the expertise of the people working on it. We had Data Scientists and Java Software Developers who knew what it was like to deal with Big Data. They had a highly performant method for gathering data and were quite good at running Apache Hadoop-based analytics. This was seldom the case back then but happily allowed me to look at less turnkey applications and assume the use of important Apache tools.

We agreed on a plan and the basic building blocks including a method for comparing results. I did the research and proposed several candidates for the tests including the Apache Mahout recommenders. It was pretty easy to rank the recommender engines we had and do some exploration of parameter tuning and choices to get our best "challenger" results. The nice thing is that we beat the old threadbare in-house recommender by a significant amount (12%). The winner was the Apache Mahout Cooccurrence Recommender using the Log-Likelihood Ratio as the core cooccurrence metric. This even though we had tested against several Matrix Factorization recommenders, including Mahout's.

We need something new

Up till this time I was only a user of Apache projects (discounting a few minor code contributions) but what I found in all recommenders we studied is a fundamental problem that is still mostly unsolved today. We had data from a retailer that included user "buys" but also 100 times more user "views". None of the recommenders could deal with this multimodal data. I consulted the authors and maintainers of the Mahout recommenders and several others we had targeted. We got some suggestions added them to our own ideas and set out to test them. For various reasons, that are beyond the scope of this post, none of the easy solutions helped and actually produced worse results so I had fulfilled the contract and left with a feeling of unfinished business.

One of the mentors of Apache Mahout, Ted Dunning, had suggested a new idea during this time. There was something about it that seemed very intriguing. He had proposed a way to use one type of user behavior to predict another. This was an aha moment for me because it codified intuition. I remember the first time he wrote in email on the Mahout user mailing list the equation that crystallized it all. I began to imagine the implications; all sorts of new data that could be useful, not just "views" but contextual data like location, and enrichment data like tag or category preferences. These all seem to obviously have a bearing on recommendations but now we had a beautiful simple equation to test the intuition.

Becoming a Committer

I set out to hack the Mahout Cooccurrence Recommender to become a Correlated Cross-Occurrence (CCO) recommender. But without some way of testing the algorithm and code we couldn’t be sure it was worth including in Mahout. The datasets publicly available at the time did not have the kind of data we needed (there had been no direct use for it until then) so I scraped the film review site rottentomatoes.com to collect "fresh" and "rotten" reviews of movies. This gave us two different behaviors with very different meanings. Naively you might think, weight one positive and the other negative and so did I but that produced worse results than ignoring the "dislikes". However when I ran cross-validation tests comparing the Mahout Cooccurrence Recommender using likes only, to CCO using both user actions, we got some quite interesting results. The question was: do "dislikes" predict "likes" and when I got 20% lift in predictive precision we could conclude that they do. Not only was intuition right but the new algorithm could tease out the data to make use of it.

The hack was accepted into Mahout Examples and I was invited to become a committer. Then the world changed.

Apache Spark and Mahout-Samsara

When I became a committer Mahout was written on Apache Hadoop MapReduce in Java (as was my hack). But it had also become obvious to most Mahout committers that the future was with much more performant engines like Apache Spark. Committers Dmitriy Lyubimov and Sebastian Schelter had been working on a Spark version of Mahout. In an instant of project time virtually all committers saw this as the future of Mahout, if also a major pivot.

In retrospect I'm not sure I've ever seen an Apache project change so much in so little time. Today Mahout is deprecating lots of old Hadoop MapReduce code as it falls from use and the new Mahout is truly new. The Mahout subtitle Samsara, references the cycle of life, death, and rebirth in the Hindu tradition. Mahout started as algorithms written specifically for MapReduce, now Mahout-Samsara is a linear algebra DSL in Scala used to roll-your-own algorithms but with most interesting algorithms in very simple DSL-based implementations. Mahout eventually took this transformation even further to include other compute engines like Apache Flink and is now running on GPUs. But I get ahead of things...

Those were exciting times and though I helped with the DSL I remained fixed on implementing CCO, which was first included in Mahout 0.10.0 in October 2014.

PredictionIO

Now we have the CCO algorithm implemented on modern compute engines but several other problems remained in order to actually deploy a recommender. This is because CCO creates a model that needs to be deployed on a special type of server that computes similarity in real time. In Machine Learning terms this is a K-Nearest Neighbors engine, known in concrete terms as Lucene, or it's scalable server derivatives like Solr and Elasticsearch. A turnkey recommender also requires a highly performant massively scalable DB, like HBase. Putting these together we could get a nearly turnkey recommendation server that made use of multimodal real time user behavior. But I didn't see a candidate for all these in Apache and so looked elsewhere. This required an integration project, not Mahout, which integrated with other services but provided none of its own.

I found a project that included everything I needed and was Apache licensed but was run by a small startup called PredictionIO. They had a Machine Learning Server that was a framework for Templates that could implement a wide range of Algorithms. The Server also included nice high-level integrations with Elasticsearch (Lucene server), Spark, and HBase. In May of 2015 I had the first running CCO Server build on Mahout and a whole list of other Apache projects.

Back to Apache

PredictionIO was at the right place to get swept up in a major move to embrace ML/AI by Salesforce Inc. who bought them as part of the Einstein initiative. Since PIO was Apache licensed OSS it was still available and so was the Template I was calling the Universal Recommender. But there was a question now about the future of PIO; what would Salesforce do with it? The old team, that I had worked closely with, wanted to see the project move forward in OSS and Salesforce seemed to agree, but large corporations often have a mixed record in promoting their own OSS projects. In this case Salesforce decided to remove the question by submitting PredictionIO to the Apache Incubator.

The old team was joined by people like me from outside Salesforce to create a project that follows the Apache Way and is free of corporate dominance. I am a committer to PredictionIO, which has three releases under Apache Incubator vigilance and the Universal Recommender is now at v0.6.0, the most popular of PredictionIO Template Algorithms.

With the 3rd release of PIO from Apache we are now in the process of graduation to an Apache Top-Level Project, hatched by the Apache Incubator. I fully expect that we'll be celebrating soon.

Postscript

My journey began with a specific problem to solve. Each step to produce the solution has led back to Apache in one way or another, through mentors, collaboration, use of, and commitment to several projects. But I now have my mature scalable, performant, state-of-the-art nearly turnkey Universal Recommender. Now we can ingest and get improvements from many types of behavior, enrichment data, and context--using it in real time to serve recommendations subject to robust business rules. My small consulting company ActionML actionml.com now has a powerful tool to solve real problems and we make a living (at least partly) by helping people deploy and tune it for their data.

This is a story of someone single mindedly following a goal over several years. There are many ways to do this in the Software Development world, but not all OSS projects are open to bringing people in. The Apache Software Foundation most certainly is and openly recruits as diverse a group of committers and members as possible. If you want to make a difference and influence the course of an OSS project Apache is a good place to look. Start by getting involved with a project of interest, make contributions, get involved in discussions. If the match is good you'll be invited in as a committer and move on from there. I think of Apache as a do-ocracy, if you do something of value it goes a long way towards being invited in.

References

Slides describing the CCO Algorithm: https://www.slideshare.net/pferrel/unified-recommender-39986309

IBM DevWorks Post on "Making one thing Predict Another": https://developer.ibm.com/dwblog/2017/mahout-spark-correlated-cross-occurences/

Apache Mahout CCO Implementation: http://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html

Apache PredictionIO: http://predictionio.incubator.apache.org/

The Universal Recommender Template: http://predictionio.incubator.apache.org/gallery/template-gallery/

Professional Support for the Universal Recommender: http://actionml.com/universal-recommender

# # #

"Success at Apache" focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be 8) Meritocracy. https://s.apache.org/DiEo 9) Lowering Barriers to Open Innovation https://s.apache.org/dAlg

Posted at 10:01AM Oct 02, 2017 by Sally in SuccessAtApache | |

Tuesday September 05, 2017

Success at Apache: Lowering Barriers to Open Innovation

By Luke Han

Over the past decade, I was a Java developer using many Apache projects such as Tomcat, Jakarta, Struts, and Velocity. In 2010 I stepped into the Big Data field and started to actively participate in Apache projects, and became an ASF Member 3 years ago. In addition to being the VP of Apache Kylin, I helped projects such as Apache Eagle and CarbonData move to the ASF, and have been a mentor for Apache Superset, Weex, and RocketMQ. Today, I'm co-founder/CEO of Kyligence (prior to that, I was Big Data Product Lead of eBay, and Chief Consultant of Actuate China).

Apache Kylin, as its name may suggest, originated from China ("Kylin": A powerful yet gentle fire-breathing creature in eastern mythology. Also written as Qilin. "Apache Kylin": OLAP on Hadoop, capable of analyzing petabytes of data within seconds http://kylin.apache.org/ ). I started this project with a few members in early 2015.

As a pioneer of the first highly-recognized Apache project from the Eastern world, I was proud to see that, within 2 years, Kylin has helped over 500 organizations across the globe to solve their Big Data challenges.

Before Kylin graduated from the Apache Incubator, the Kylin team faced a lot of cultural challenges. Since a great number of projects from China had failed in the past, we too received many questions and doubts from both eastern and western worlds. As our native language is not English, communication with mentors did become difficult during the coaching process. Fortunately, by fully embracing The Apache Way, Kylin is able to succeed with strong support from the Apache community members. Much more beyond the Kylin software, our team has also worked with those talented people in a way to spread our Chinese voice to the world.

While developing high-quality software, we are engaging more Westerners to understand the Eastern culture. I had many chances to travel and meet people across the globe since I initiated Kylin. Some of them are Apache directors and mentors, some of them are developers and contributors. Some are from US, Australia, Canada and Chile; some are from Japan and Taiwan. Some are impressed with Kylin, some are curious about Easterners’ attitude toward Open Source software. I asked them a lot of questions about The Apache Way, and they all generously coached me and my team with lovely and detailed answers. We too could reach consensuses after intensive and open arguments. Kylin received much more encouragement and recognition than I expected.

As a VP of a Top-Level Project, my responsibility grew after Kylin graduated from the Apache Incubator. Kylin faced more opportunities as it has been bug-fixed quickly and tested frequently, with the nature of an Open Source software. In the China’s well-knowingly-big market, Apache Kylin has received many users’ feedback and evolved fast. We received many suggestions from both developers’ perspective and products’ perspective. Beyond my expectation, many community members are passionately writing tools for Kylin and helping users better understand and use Kylin. Assembling members’ ideas, we are also sharing our knowledge as a way to give back to the community.

Thanks to ASF and everyone involved in the Open Source community, I have the opportunity to work with people that I’ve always admired and make a difference in the world all together. I feel I and my team are deeply connected with such warm, global, open community.

= = =

# # #

Posted at 12:45PM Sep 05, 2017 by Sally in SuccessAtApache | |

Tuesday August 15, 2017

Success at Apache: Meritocracy.

By Kevin A. McGrail

The Apache Software Foundation is not a democracy.

It's an elitist organization that does not support an innate right to vote. We aren't capitalists because you can't buy a seat on our board. We aren't socialists since we place building working communities over software. Monarchy doesn't fit because Kings and Pawns work together as equals.

What we are is a Meritocracy. To be able to have a say, you have to prove your worth in a system of merit. Meritocracy is a key part of The Apache Way. With it, the ASF creates amazing software with amazing people that continues to change the way the world computes.

Merit has no basis on Age, Sex, Religion, Ethnicity, Race, Country of Origin, Sexual Preference, Social Status, Income Level, Lineage, and/or Physical/Cultural Traits*.

In honor of The Apache Way, the ASF has created two wristbands to share with the tech community. The first is silver and announces our Meritocracy.

The second, because merit is NOT rooted in biological differences, is brash and bold in Red announcing "do I.T. like a girl". The idea comes from Code Like A Girl wristbands coupled with a small bit of double entendre to start a conversation about improving inclusion.

If you'd like some wristbands, they'll be debossed in a single color like the pictures below. Just send me an email at kmcgrail(at)apache(dot)org. I'll try and send out as many as I can for free. If you like/hate the idea, feel free to send me an email as well and tell me what you would do differently.

* NOTE: We do take into serious account whether you are a Cat or a Dog person.

Kevin A. McGrail is a cybersecurity expert and Open Source advocate who loves stopping spammers. He got involved with the ASF when the Apache SpamAssassin project joined the foundation in 2004. Today he still helps the SpamAssassin project while also serving as an executive officer and VP of Fundraising.

= = =

"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works". 1) Project Independence https://s.apache.org/CE0V 2) All Carrot and No Stick https://s.apache.org/ykoG 3) Asynchronous Decision Making https://s.apache.org/PMvk 4) Rule of the Makers https://s.apache.org/yFgQ 5) JFDI --the unconditional love of contributors https://s.apache.org/4pjM 6) Meritocracy and Me https://s.apache.org/tQQh 7) Learning to Build a Stronger Community https://s.apache.org/x9Be

Posted at 01:06PM Aug 15, 2017 by Sally in SuccessAtApache | |

Monday June 05, 2017

Success at Apache: Learning to Build a Stronger Community

by John Ament

As the next line in the series of "Success at Apache", I had to think about what kind of blog post I wanted to write. Given my personal focus, it made sense to focus on new projects coming in and the incubator. When I'm not busy dreaming up new ideas and working on personal projects, I'm helping new projects get in to Apache, keeping their goals in alignment with the Apache Way http://apache.org/foundation/governance/ . I'm a member of a few different PMCs here at Apache, notably the Incubator. I'm a mentor to five different podlings right now. While my primary programming focus is on programming models, my podlings are all over the place. Starting a new project here at Apache can be a daunting task: how do I get in? What if I don't build a diverse community? Becoming a podling has more to do with the community than it does the technical aspects of the project. We don't expect you to be experts in it, but we do expect new projects to be experts in how their own software works. We want to teach you, and we want you to be receptive to learning about The Apache Software Foundation and its best practices.

I'm not sure if everyone does it, but I build a lot of parallels between how an ASF project works and how an Agile team works. Agile teams start off as a bunch of people who don't really know each other but have assembled themselves into an informal team focused on solving a problem, or some number of problems, knowing that they can only do it together. They have common goals and objectives, but lack camaraderie early on to be able to work together smoothly. Over time, they get to know one another, figure out strengths and weaknesses and can resolve issues together. A well-functioning team isn't one at the beginning. It takes time and practice for them to work well - both together and as an outwardly facing unit.

Projects here at Apache follow the same type of maturity progression. Whether it's learning The Apache Way or learning to work with one another, it takes them time to mature and get into a good groove.

Open Communication
The ASF is pretty big on open communication, wherever it's a sensible solution. We want to discuss with each other what we're doing, ideas around how to solve it and come up with a good solution together, as a team, in an open manner.

This all ties into agile practices. We host stand ups to talk about what we're doing and see if others have an opinion about what we're doing.

When a project comes to Apache, the original authors need to remember that they're bringing in a lot of experience, and the expectation is that those existing contributors must help get new contributors from the outside - outside their organization specifically - to contribute into the project. By driving towards open communication, outside of your own organization, you're encouraging more people to participate. This sort of governance model ensures that all parties who can participate are aware of decisions being made.

Open Communication isn't for everything though. We need to remember to be respectful in our communications with others and if it's felt that something’s awry - speak privately. But remember that isn't part of the decision making process. Likewise, anytime we're talking about individuals in either a positive or negative way that should be conducted on the private list for a project.

Turning Into a Well Oiled Machine
Once a project begins to grow, new people start to get attracted to it. As a community, you have to figure out how to work together. Building a community of diverse ideas and skills will ensure that new ideas keep flowing. Contributors can react quickly to a user's question on list and help them resolve the problem, put in an enhancement request or get a bug report squashed in a following commit. Time is of the essence right now because I have availability to work on this.

There can't be a long drawn out waterfall style process when dealing with Open Source. At the same time, making sure there's a documented decision process and in sometimes an in depth design is critical for both new contributors and existing alike to come to a shared understanding of what is being proposed.

Sustaining
Projects need to plan for longevity. Longevity comes in many forms. A strong backlog of features is important. Having a diverse set of committers is even more critical. You could even say that each helps create the other. Just like any feature set, we get to a point where the feature is complete enough that we can move on to another feature.

How do you get there?
Apache's main way to go to these points is to incubate http://incubator.apache.org/ . You can't get to this point by yourselves, experiencing with first-hand from existing Foundation members will help get your community to turn a new leaf and adopt this way of working. We want you to be successful, as long as your project can dedicate itself to the practices that have been set forth within the Foundation.

New projects may be comfortable with a champion http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion that can work with them closely, answering their questions up front. While a lot of the pre-incubation chatter will happen off list, it is important that potential new podlings subscribe to the incubator general list http://incubator.apache.org/guides/lists.html#general+at+incubator.apache.org and understand both the goings on of a podling as well as try to build their list of mentors http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Mentor in the open. Mentors are extremely important to a podling, and understanding their roles and why you need to pick great mentors is something your champion and the rest of the Incubator community can help explain. Participating in our public discussion lists is sometimes the first step to joining the foundation at a deeper level.

Where do we go next?
If you're a potential new project, feel free to reach out on the Incubator mailing lists http://incubator.apache.org/guides/lists.html#general+at+incubator.apache.org to get started. We'd love to hear from you and get you acquainted with The Apache Software Foundation.

If you're on an existing project, we want to hear your perspectives on how the Foundation works. You may want to reach out to dev@community http://community.apache.org/lists.html to let others know your thoughts, or even just subscribe and see what others have to say. We're all working together to make the foundation better. The more input we receive, both the positive and the negative, will help shape everyone's actions in the community.

= = =

Posted at 12:04PM Jun 05, 2017 by Sally in SuccessAtApache | |

Monday May 01, 2017

Success at Apache: Meritocracy and Me.

by Tom Barber

When Sally asked for volunteers to help with a blog post series "Success at Apache" I realised there was a very human story to tell about how the ASF helped me get to where I am today and hopefully where I'll go tomorrow. Over the years I have worked on and run a number of Open Source projects whilst working with an awful lot of Open Source software. One day I was browsing Slashdot as you do, yeah I know a lot of people disparage it, but it's an awfully hard habit to kick, and without it I wouldn't have got involved in the ASF so I owe it a lot. Anyway, one day when browsing Slashdot I saw this article (https://it.slashdot.org/story/11/01/08/1544204/apache-to-steward-nasa-built-middleware), I had been working in the Open Source business intelligence industry for a few years at that point and I spent a lot of time hacking around and managing data systems, so I wondered how I could get some help out of OODT (http://oodt.apache.org/). Also as a kid I had always loved everything about space, I was a huge Apollo fan, had a small telescope, went to the total eclipse in the UK in 1999 and so on. I thought this OODT project would be a fun way for me to chat nonsense to a few NASA employees, find out how they did stuff and do a bit of Open Source hacking on the side, which would at least let me participate in some NASA related development work, and so it began.

For those of you who haven't heard of Apache OODT it is a middleware layer for building data systems. Originally written by NASA JPL and then Open Sourced to The Apache Software Foundation it provides data ingest capabilities, metadata extraction, data workflows and resource management. I started by asking pretty dumb questions on the IRC channel, posting stuff on the mailing lists and trying to figure out how this reasonably expansive stack of software even operated. Chris Mattmann and Sean Kelly guided me through the opening foray into OODT development and education. Eventually, having submitted a few bug fixes, I volunteered to be a release manager for an OODT release and that got me more involved. Not too long after Sean asked if I fancied having a go at being the project chair, which I duly accepted. Behind the scenes cogs were turning and in a matter of a few weeks, I'd gone from a committer and PMC member to Chair to ASF member, it was certainly a hectic time, trying to keep up with all the new things I had to do, mailing lists to follow and so on, but what a period in my ASF experience, lots of fun!

That was just over 2 years ago and I'm still happily stewarding the OODT folks and keeping the cogs turning, releases happening and Jira tickets triaged. Alongside that it is truly an honour to be involved with the ASF as a member and although the politics can get tedious, the foundation is an amazing place for people to learn to work on great software as part of a distributed team. 18 months ago I was getting a little jaded with the monotony of the BI work I was doing, there are only so many sales databases and budget reports one guy can take and after being in BI 8 or so years I felt like it was time for a change of scenery, I just didn't know what. So I blasted out an email to 10 or so people I knew or had had some contact with over the years who might be able to give me a job, or know someone who was looking for a Java developer, BI guy, Open Source advocate, that type of thing. I'd included some OODT folks on my email, not because I thought there was a chance of a job using it, but just in case they happened to know someone out in California needing some remote help. Everyone said no, except Chris Mattmann who said if I could hold on a few months he might have something for me at NASA! That response floored me, I'd never even considered that as an option and knew that with a young family it would be highly unlikely I'd be able to move to California, so I played along assuming it would fall through. But the as the process dragged on and contracts got drawn up we got closer and closer to it becoming real and the excitement grew, there was the tangible possibility of me fulfilling at least a bit of a life long dream, no I wouldn't be an astronaut, but there was the chance of employment by NASA.

Eventually 6 months or so later, the paperwork was signed and I joined the ranks at NASA JPL, working as an Apache OODT and devops guy. What is great is that having 10 years of business and development experience, I feel like I can very much make a positive contribution to the team, and in part that is down to what I have learnt developing and coding at the ASF. It has been an amazing experience and a wonderful 12 months. I never dreamt an opportunity like that would arise and it is 100% down to the great work the ASF does in stewarding new projects through the incubation process and into mainstream adoption. Without the ASF I would likely still be a BI guy dealing with run of the mill data warehouses, instead I work on Genomics Search Engines, help hunt criminals on the dark web and a host of other stuff. Life sometimes throws you an opportunity that you don't expect and the ASF certainly facilitated that.

Last week I was in Pasadena, visiting the JPL facility and getting the guided tour, and doing a bit of work. It was amazing talking to such a dedicated group of people who clearly have a big passion for what they do. Getting to see their mission control, the mars rovers and various satellite mock ups was awe inspiring but what excited me the most was getting to sit down and pick the brains of people with whom I have worked with at the ASF for years yet not met in the flesh. Finally making that human connection means a lot.

What the ASF offers here is the ability to learn to work as a distributed team without the pressures of the "real world". Everyone at the ASF, pretty much, is a volunteer and other volunteers recognise that, and so it reduces the pressure, but whilst reducing the pressure it teaches you how to make binding decisions as a disparate group, how to keep records and how to ship good quality code whilst living in different timezones. At the ASF some of us might meet once or twice a year at ApacheCon, Fosdem or elsewhere, but largely all communications is done via mailing list. This can cause issues when people "just want to get it done" but it also provides an immutable record of what is going on in a project and who said what. This proves equally useful out in the "real world" where you want to track business decisions or look up historical records of why a certain choice was made. Also dealing with people who you don't work with on a daily basis also helps you think more about your communication style, what is fine to say and what isn't and also how you structure your communications, which is also very important in a business setting. Do you know the person? Do they understand your nuances? Is English their native language? etc

The other thing I find the ASF offers is understanding. Last year I was diagnosed with Aspergers at the age of 33, which is pretty late. What is nice is that generally, people like to listen, and if you have something that affects your personal or professional life, people who you've met at the ASF will often lend an understanding ear to allow you to off load or discuss something completely unrelated to the project you might be working on. Or like me, you can just stand up at the front of an ApacheCon lightning talk and tell everyone! Either way, you can generally find someone in the Apache family who will provide a sounding board for anything you want to discuss.

These days I spend my spare time still working on OODT stuff, but also doing a lot of public speaking and mentoring and whenever I do I make sure I talk up the Apache Software Foundation because it has given me the chance of a life time and one that I'll be forever grateful for. If you aren't involved in development here at the ASF, get involved, you don't have to be a coder, you just need to like helping out in a fun, Open Source community.

As I mentioned at the start, this blog series is about success at Apache, hopefully this proves that success can come in a number of ways, the ASF was selected by NASA as the home for its data middleware platform, that proves that the NASA deemed the incubation process, the license and ecosystem acceptable, that is success the the Apache Foundation. Similarly the foundation has proved very successful in placing people into employment from a range of different walks of life into new lines of work, and that is exactly what happened to me and the reason I wanted to share my story about success at Apache.

= = =

Posted at 01:05PM May 01, 2017 by Sally in SuccessAtApache | |

Tuesday April 04, 2017

Success at Apache: JFDI --the unconditional love of contributors

by Daniel Gruno

In many respects, The Apache Software Foundation is like a dog.

Some people like dogs, some like cats. The ASF doesn't care, it still loves you. Some people are female, some are male, some don't share the narrow dichotomy of the masses. The ASF doesn't care, it still loves you. Some people have cool parents, and a car, and money, and are connected with the right friends, and some are "strange" loners that can count their friends on one hand (sometimes you don't even need a hand!). The ASF doesn't care, it loves you anyway...as long as you feed it some code or documentation (or any of the other valuable contributions), you get love and respect in return.

But before I start telling my story about a dog named Apache, I should probably tell you a little about myself. I've been involved with Apache (both the foundation and the good ol' HTTP Server) for five years now. I am a Member of the foundation, and the Chair of the Apache STeVe project. The reason I got involved and why I'm still here, working for the ASF as an infrastructure architect, is that I had an itch (several in fact) to scratch and a yearning to show the world that I could...do stuff!

I came to Apache from an academic background. I had been studying at various universities in Denmark, initially Statistics and Business Administration, and later on Human Resource Management, so my assumptions about how Open Source worked were that it probably worked like academia works: You have an idea, you present it, ask for feedback, start a collaborative process and have your peers review it, then implement said idea once they all say it's okay. Now, in some regards, this is accurate, but the method of execution is vastly different.

Academia is built around a mix of healthy and unhealthy inherent mistrust (in many respects akin to the notion of original sin in some beliefs).

The healthy part was in my experience --and in very(!) simple terms-- attributed to critical thoughts of Karl Popper and like-minded science philosophers who, argue against _proof_ as a means of advancing science, but rather seek out a _lack of evidence against_ as the proper way to corroborate a theory (and in doing so removing things like "is there a god?" or "are gnomes real?" from the sphere of academic theories, as you cannot disprove the existence of said figures, thus they are categorized as meta-theories and philosophical conundrums instead --the question of "do we really deserve dogs?" however is still valid!). Instead of proving that the sun does indeed rise every day, one would instead say "I have a theory that the sun won't rise tomorrow" and then debunk that. While proving that the sun rises tomorrow is a nigh impossible task (even with a 99.999% chance of it rising tomorrow, that means there's only a 16% chance of it rising in 500 years time), proving you were wrong tomorrow when the sun rises yet again is a simpler and more attainable goal that _has practical value_.

Now, while this works well in academia, the notion of having to _prove your code doesn’t work_ can be a bit much for someone just trying to fix a typo in a script. Still, things such as unit tests and fuzzing can be compared to the notion of _proving by failing to disprove_, in that we too in Open Source employ a notion of "if we can't break it, it must be working". This calculated and practical mistrust in our work is healthy.

The unhealthy part stems from our fellow human beings being human beings...and not dogs. Unlike Open Source communities like Apache, universities are, in my experience, just high school all over, with fancier charts and bigger books. It matters who you know, how well you can network, who foots the bills. While some schools do make you feel at home, in most cases you are left to fend for yourself and build up a network, both socially and scientific. If you did not have the proper social skill-sets, you were alone. There was no sense of inherent belonging, it was --in a sense-- the capitalist dream turned to a nightmare. Your future was yours to create, and yours alone. If you lacked the skills or mental capacity for socializing, you were simply left behind.

A Dog Named Apache
Looking back at my experiences with education institutes, imagine my surprise when I --the classic loner type with limited social skills-- asked if I could get some patches applied, and five minutes later, they were! The response was, to me, an overwhelming acceptance of the work I had done, and people saying "please do contribute more, we value this immensely". More than that, it was a sense of being invited to a community that didn't have any other reasons to invite me than "I have some patches".

At Apache, it didn't matter who you had known for years, or what your social standing was, what you looked like or any other generic measurement we generally use in the outside world. If you have something to contribute, and it makes sense, you are welcomed with open arms.

From the very beginning, I was gently nudged to contribute what I thought was interesting --not what THEY thought they needed, but what I had an interest in solving-- and I saw a profound interest in my skill-set and ideas. I saw people thinking what I did was cool, and I was cool, even though they had no idea who I was. It was like suddenly making friends in a platonic speed-dating séance where all that matters was being interested in doing SOMETHING --didn't matter what it was, as long as it made sense.

There is a general notion that Apache is a meritocracy: You contribute, and through your contributions earn merit that in turn affords you leverage and say in matters. I'll posit that not only is this true, but it's also a positive sum game with "original merit" applied. People are inherently trusted to have good intentions and are afforded an initial goodwill that you might not afford people in other circumstances. I didn't know these people, they didn't know me --and none of that mattered here.

Fast-tracking ideas

Within a week(!) of contributing patches to the HTTP Server project, I was voted in as an Apache Committer, much to my surprise. Even more surprising was the attitude at Apache, especially the Infrastructure Team: If you want to do something, go ahead and do it (with minimal supervision). Here's a server you can hack on, here's a place you can put your code, here's someone who will help review it! I had an idea of making a comment system for the HTTPd documentation, and (again) politely asked if it was at all possible that I could write it. At this point in time, I was expecting a bureaucratic NO, with some explanation of how they didn't know me (and thus, why would they entrust me with their hardware?). The answer was a very terse "JFDI, here's a FreeBSD jail for you". While I felt a bit scared of the general tone at the time, the notion that you could just do something without having to spend time gaining trust, requisitioning things, getting reviews prior to implementation and so on, was exhilarating to me: I could hack on something, I HAD A BOX TO EXPERIMENT ON, no strings attached! Again, the notion that you were inherently trusted was present. It didn't matter that I hadn't worked with infrastructure before, I had an idea that could solve a problem, and to them, that was all that mattered. Welcome to the team!

And so I wrote a comment system for our documentation. It got implemented in the documentation, other projects saw it and said "can we please use this too?". Not long after, I was neck deep in infrastructure business, having discovered that Apache was not only the HTTP Server...It was a plethora of interconnected projects all sharing the same notion of coming together to solve problems and help make the world a better place through advancing computer science. Everywhere I looked inside Apache, the notion seemed the same: If you can help us, you're one of us. Don't care who you are, where you come from, as long as you can contribute in some way, you're welcome in our community as a valued member.

Fast-forward
So I joined another project, and another, and another. Today I am an official part of 10 Apache projects, on the Project Management Committee (PMC) in 8 of them. Do I know about these projects in extreme detail? Heck no! I can barely tell you what some of them really do, but that really doesn't matter at Apache. What matters is your willingness to contribute, no matter what your expertise level is, no matter what field of expertise is. If you have something to contribute, Apache will accept and love you. Just like a dog.

So put down your phone, stop facetweeting, and most importantly, stop thinking you can't possibly help or become a part of Apache. If you can write an email, you can help out. If you can fix typos, you can help out. If you can *sort* of code in a programming language, you can help out. If you can write newsletters, know how to fix a configuration error, help people on IRC, you can help out...and Apache will love you for it. And before you know it, you'll be a deeply integrated part of the Apache community.

If you are in doubt as to what project you would or could contribute to, Apache has an awesome Community Development project that helps mentor and engage people in projects, as well as teach about how the foundation and projects work. For more information, head on over to https://community.apache.org and check out the resources available to you. You can also check out https://projects.apache.org and see if you know one of Apache’s projects, or perhaps discover a new project that fits what you are interested in --welcome aboard!

# # #

"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works":

1) Project Independence https://s.apache.org/CE0V
2) All Carrot and No Stick https://s.apache.org/ykoG
3) Asynchronous Decision Making https://s.apache.org/PMvk
4) Rule of the Makers https://s.apache.org/yFgQ

Posted at 03:07PM Apr 04, 2017 by Sally in SuccessAtApache | |

Monday March 06, 2017

Success at Apache: Rule of the Makers.

By Nick Kew

I started working on a range of Web applications in the 1990s, the first of them internal to my (then) workplace where it provided an operator interface to the daily processing, archiving and distribution of satellite image data; the second a forerunner of what is now called social media, and my first use of an Apache server. The release of Apache HTTPD 2.0 drew me from server user to developer: in part because I needed to re-implement some existing functions, but more excitingly because I saw tremendous potential for the server itself to become a powerful applications platform. This led me to working on the core software and interacting with the Apache community alongside releasing my own modules and documentation. In 2003 I gave my first ApacheCon presentation, and sometime after that was invited into the Foundation first as a Committer, and became a Member in 2005. Since then my interests have encompassed not just the Web server and related projects, but also the Apache community and its dynamics. I’ve been involved in mentoring several projects through the Incubator. If you were to ask me today about the single goal I’d most like to accomplish, it’s a framework for Identity management that is not merely cryptographically strong, but sufficiently straightforward for the world to use, and robust against social engineering attacks such as phishing, while at the same time free of any centralised authority (such as government) whose motives might come under suspicion. An end to identity fraud, and to password management nightmares.

Much has been said and written about The Apache Way [1]. It typically focusses on the importance of community, and on the democratic and meritocratic elements of project governance. And on the role of an Apache project's formal governing body, the Project Management Committee (PMC), comprising contributors elected by the community on the basis of their track record of contributions and constructive engagement.

In practice, the role of the PMC is largely reactive. The big, interesting questions like "what direction does our project go from here" are discussed in public, where inputs from the wider community are welcomed. A lot of the nitty-gritty detail is determined by the needs of the community as measured by feedback received and seen, and by the needs of individual developers. The latter is often described as "scratch your own itch".

Setting aside the purely reactive, the detail of what actually happens is often determined as much by what a developer is prepared to work on than by any grand plan. If you do the work, then you control what happens. If you need something, you go ahead and make it happen. This has occasionally been described by a clumsy English-Greek hybrid word "do-ocracy". In the hope that it's not too late to nip that in the bud before it becomes as ubiquitous as the Greek/Latin hybrid of which CP Scott memorably told us no good would come [2], I shall instead call it Pratocracy. Rule of the Makers. It applies both within an Apache core team and in the broader community.

So how does a Pratocratic project work? How does it avoid the chaos of a Tower of Babel? How does it remain cohesive and focussed? And if an Apache project looks like a good basis for your company's needs, how best can you work with the Pratocracy towards your goals?

Taking the first question first, how does the Pratocracy work? Eric Raymond proposed the contrast of the Cathedral vs the Bazaar: the autocracy of the traditional project vs a free ecosystem. The Bazaar ecosystem is essentially Darwinian, in that the strongest projects prosper and evolve, while a Long Tail go nowhere very much and are forgotten as their authors move on to new things.

The Apache processes are all about ensuring our projects are fit to be among the winners in that ecosystem. An Apache project is a graduate of an incubation process [3] that requires it to build a broad development community, with sufficient diversity to survive the loss not merely of a key individual, but even of a key company team that might have been the originators of the project. The key principle is that if the community is healthy, the software will follow.

Thus the context of the Pratocracy is selective: it is the well-known Meritocracy element of Apache. Members of the core community are elected based on demonstrated commitment to the project. That means they understand the distinction between Apache work and company work, and are comfortable holding multiple allegiances and avoiding or resolving potential conflicts between them.

So, to the crucial question. Suppose I have identified an Apache project that doesn't fully meet my company's needs, but is the ideal starting point for our project. How do I best work with the Pratocracy to make it happen?

Let's consider some possible approaches:

Ask via the development list and issues database.
Approach developers individually to discuss the project.
Just go ahead with the company's in-house developers.

Each of these approaches has merit in its own right. But in many cases, the ideal approach will encompass all the above. In an Apache community, the whole is greater than the sum of the parts. The closer you can work with the community, the more benefits you stand to derive, particularly if your project finds itself wanting to integrate more deeply with the Apache project than is provided for in stable public interfaces, when the alternative to working within the Community is a patched core and the risk of a maintenance nightmare. On the other hand, there are good and bad ways to engage with an Apache community: the right approach can make all the difference to the benefits you can derive from it.

First, it's always worth asking about your needs. Post a question about your requirements to the project's lists, or issues database. Engage in discussion. It may be that someone somewhere has already done the work, or that it's closely aligned with someone's work-in-progress. Even if you draw a blank, it's a start to engaging with the community.

Approaching individuals is most likely to be appropriate if you're looking to hire in expertise. General questions are likely to be redirected to public support fora, but some Apache developers work as consultants and may welcome a professional approach. They can help you figure out the best way to proceed, both technically and in terms of working with the Apache community. But bear in mind the potential for conflicts of interest. If you want your work to happen within the Apache project, your consultant will want to be satisfied that you and your project are indeed a good fit, and won't cause trouble such as unmaintainable software down the line. And of course, any NDA should make it clear you're not laying claim to the consultant's regular Apache work!

If you have the in-house developers for it, you might just task them to go ahead with the work. That's easiest when you can work with public, stable interfaces of the Apache project, harder and messy if you have to hack into the core. When you start that work, the Apache community becomes your team's best resource, so if your developers are any good (and not cowed into keeping their heads down and never talking to outsiders) they'll be interacting with the Apache community as they go along.

Now you're participating in the Pratocracy, albeit (usually) outside the core community. You have something to offer. If you share it with Apache, members of the Apache community will take a look, and may get involved. At best this can lead to a virtuous circle where your developers not merely benefit from the Apache team, but contribute to it, and earn themselves the merit to be invited to join. But this is by no means automatic: both developers and their managers need to be able to see outside corporate control structures. Crucially, individual developers have to be comfortable separating out their Apache hats from their professional allegiances, which means managers have to allow that to happen. But don't try to manage such a process from above: that's herding cats. Just facilitate!

That probably means that at some stage you'll have to sit down with the team and fix boundaries: what belongs in the Apache project vs what belongs in your company's own space. It helps immensely with developers' confidence if they're not forever looking over their shoulders and wrestling with ambiguity. This may be obvious if your project isn't destined to be shared openly with the world, but it's a discussion worth having even if everything is Apache-licensed. In the latter case, the boundaries will focus on what your company expects to sell and/or support commercially and wants to maintain full control on, vs what passes out of your direct control.

Even if your company merely clones or mirrors the Apache project and contributes everything, you'll want to separate out matters of your business model, and customer-facing vs Apache-facing matters.

Note that it's easy to get this kind of thing wrong. Something we sometimes see at Apache is developers who've been told by their managers to join an Apache community, but have no specific tasks and no motivation. That becomes painfully obvious, and leads nowhere useful.

The right approach is to give your developers real-world tasks and let them come to the Apache community - if their course takes them that way - in their own time. If close interaction with the Apache project is an important goal, you may want to approach and maybe hire individual developers having a strong track record with Apache or comparable Open Source environments.

To conclude, let's consider a real-world example from my own experience with HTTPD (the Web server still commonly called "Apache") and APR projects. The history of SQL database integration into the Web server spans many project teams, both within and outside Apache, both corporate and community-driven. Each developer or team just went ahead with something appropriate to its time, and (at least those you or I have heard of) achieved success in their respective contexts.

In the early days, there was a Web server with no SQL support. So any application that required it had to manage its own database connections. A range of third-party modules were developed for purposes such as authentication by SQL, and logging to SQL, both with popular databases such as MySQL and PostgreSQL and in commercially-supported products with commercial databases. Generalised frameworks for Web/SQL applications were provided by scripting modules, pioneered by mod_perl, giving developers a language's own database interfaces, together with a persistent database connection.

In time, the server and its applications outgrew this ad-hoc approach, with multi-threaded processing and multiple applications in a server. Building on a range of ideas, a native SQL database connection framework was proposed and developed, offering some major efficiency gains to applications through dynamic connection pooling and sharing. It also promoted rationalisation, so in an area like authentication, a whole raft of different database-driven modules could be replaced by a single module for generic SQL authentication. Different SQL databases are handled by per-database drivers which serve all kinds of application in the server.

This work took place within the core Apache team, and propagated more widely as others such as the scripting languages incorporated it as an option for their users. It evolved from minimalist beginnings to provide more capabilities as required by developers "scratching an itch", and has benefited from contributions of database drivers from a range of different sources.

Having said that, it remains intentionally fairly small, and won’t meet the needs of every application. Some projects continue to use SQL their own way. The fact there is now a Right Way to access SQL doesn't mean that other ways are wrong! There are still third-party projects that, for their own reasons, use other ways to access SQL. The fact that someone has done the work and met a need gives them an entirely valid niche in a Pratocratic ecosystem.

The bottom line? Many Apache users can just use our software off-the-shelf (perhaps after asking around for solutions to problems), others may need to add significantly to it to meet their needs. If you fall into the latter camp, just go ahead and do it: you can discuss and/or share your work at any point you choose! The basic management decision is the tradeoff between control of your work and community input. Interaction with the project's core people is down to how your developers work best. If leaving that to chance is not sufficient --perhaps because you would like to influence the project's core software or introduce new APIs-- that's the point when you'll need to consider your developers' track records in community-driven Open Source, and make sure you have that expertise to hand.

[1] https://www.apache.org/foundation/how-it-works.html

[2] https://en.wikiquote.org/wiki/C._P._Scott
[3] http://incubator.apache.org/incubation/Process_Description.html

# # #

"Success at Apache" is a new monthly blog series that focuses on the processes behind why the ASF "just works":
1) Project Independence https://s.apache.org/CE0V
2) All Carrot and No Stick https://s.apache.org/ykoG
3) Asynchronous Decision Making https://s.apache.org/PMvk

Posted at 02:04PM Mar 06, 2017 by Sally in SuccessAtApache | |

Monday February 06, 2017

Success at Apache: Asynchronous Decision Making

by Bertrand Delacretaz

Asynchronous decision making is a key enabler of our geographically and culturally distributed Open source teams. In this post I'll explain the ingredients that make it work at the ASF.

I became active in the ASF in 2001 via Gianugo Rabellino - he was the one who started the discussions with Apache Fop about me donating the jfor XLS-FO to RTF converter that I had developed earlier. It was already too late to uninvent RTF which is a terrible format, but I digress. I am currently a member of the Board of Directors of the ASF and have been doing a lot of thinking (and presentations) about what makes the ASF tick in terms of collaboration and Shared Neurons.

If synchronous decision-making meetings were required in ASF projects, even using remote channels like IRC or videoconference, we would move forward at a snail-like pace, as just finding a time where all stakeholders are available is almost impossible in an environment that has no managers and no central schedule.

Meetings are also very expensive when you are working on a maker's schedule, as described by Paul Graham [1]. Frequent meetings ruin the productivity of craftsmen, and there's lots of craftmanship in our industry, especially when you're building leading edge stuff.

So, what's needed to enable people to make collective decisions asynchronously, without requiring meetings?

The first thing you need is a central asynchronous communications channel. Which technology you use for that doesn't really matter, but it has to allow everybody to get the same information, and provide a usable way of having threaded discussions, where you can branch off on a topic while ignoring other topics being discussed on the same channel. This can be as simple as a whiteboard if people often visit the same place, or as elaborate as web-based forums, accessible from any mobile device so you can bother^H^H^H^H^H reach people everywhere. At the ASF we use plain mailing lists for that, very successfully when people use them with the right discipline (see the appendix below). Archiving this channel is very useful, to allow newcomers to get a feel for how things work as well as documenting the reasoning that led to each decision and avoid having to repeat things over and over.

The second required tool is a way to build consensus, where you avoid deadlocks and make sure decisions go forward. Unanimity in decisions is ideal of course, but the second best is consensus, defined as widespread agreement among people who have decision power. Requiring unanimity or allowing vetoes in decisions can block progress, so at the ASF vetoes only apply to a very limited set of decisions types, as defined by our voting rules [4]. In companies, decision power can be based on hierarchy to break deadlocks. That has to happen sometimes, but abusing it can cause employees to lose their autonomy and purpose, which kills your team in the long term.

To keep track of each decision, a case management system is ideal. You could work without that, depending on your team's size and the number of decisions that you take, but it's very convenient to be able to discuss the details of a given decision and keep associated information in a single place. You don't need complex software for that, at the ASF we use fairly simple issue trackers. Those are Web-based systems where each case is handled on a single page, with a history of comments and actions. Some non-urgent or very hard decisions can take a long time to reach closure, and it's very useful to keep their history in a single place, if only to avoid having to explain them again to new members of the team. In a low tech environment you could just use a single sheet of paper to briefly document each decision with the key points that led to it, and keep those in binders or physical files.

A nice side effect of using case management software is that each decision gets a simple unique identifier, like FOO-123 for the 123th ticket of the FOO project. This removes any ambiguity as to which issue one's discussing, by mentioning those identifiers in conversations.

So, in summary, the following should allow your group to make decisions asynchronously, without requiring meeting and with a written trace of everything that happens:

An archived asynchronous communications channel, where everybody can get the same information and threaded discussions can take place.
A way of building consensus, including fair rules for breaking deadlocks.
If possible, a case management system to keep track of each decision's details, in a much cleaner way than the often messy discussions that happen on the asynchronous channel.

Semi-asynchronous decision making at the ASF

I've been a member of the ASF's Board of Directors for a few terms now and I'm still impressed by how efficient our monthly phone conferences are. The meeting regularly lasts only 60 to 90 minutes, during which we approve around 50 project reports, vote on a few resolutions and often address a few discussion items.

Besides a few simple things like good phone discipline and a side channel for less important comments (and jokes), the main reason this meeting is so efficient is that almost everything is decided in advance.

Board members are expected to read the project reports before the meeting, and a dead simple case management system (described below) helps discuss issues in advance, and find out which reports require a more extensive discussion.

Assuming the majority of board members have read the reports in advance, and flagged them as ok or requiring discussion, we don't need any housekeeping time during the meeting. Everybody shows up with a clear view of where difficult discussions might arise, so they have time to prepare for that, including asking others for clarification before the meeting so we can resolve any outstanding issues without delay.

The case management system that we use for this is extremely simple, but in terms of enabling asynchronous (or rather semi-asynchronous) decision making it fullfills its role. Our meeting agenda consists of a single text file in our source control system, with a simple structure that provides for a small discussion space for each report that we have to approve and each resolution that we need to vote on.

The agenda file structure looks roughly like this:

Call to order
Roll call
Officer reports
Project reports, headers and discussion space
Board Resolutions with discussion space
Appendix: Full Project reports and other supporting material

And a project report header and discussion space is as simple as this:

E. Apache Blazinator Project [Bob Blazer / Bertrand]
See Attachment E
[ Blazinator.
approved: bd, mm, dd, db, jc, ldv
comments:
bd: Not sure why LEGAL-123 blocks their release
ldv: They are waiting for the committer to supply
an updated iCLA as the received one was
incomplete.
bd: Ok, thanks, approving the report then.
]

This simple block of structured text builds a very simple "case management system" for the case of approving the Blazinator report.

The "approved" line indicates which board members have approved the report, on a single line so that simple text-based tools can validate and count the approvals.

The "comments" section allows stakeholders to comment on the report (which is found in an appendix later in the text file), and reply to each other's comments to hopefully reach closure before the meeting. If this happens, approving this report takes almost no time in the meeting, the chairman can just list the project names ("case identifiers" according to the above terminology) of such pre-approved reports, asking if anybody's opposed to approving them.

Combined with the ASF board's mailing list, this builds a very simple and very efficient system for semi-asynchronous decision making. Most decisions are taken before the meeting, and the participants can spend their time where it actually adds value as opposed to exchanging boring status information during the meeting.

Try it yourself!

Many ASF and other Open Source projects release world-changing software while having no or very few meetings, demonstrating that these techniques work.

If you're bogged down with inefficient or useless meetings, I suggest that you try applying these principles to a meaningful subset of your decision making activities. People will need to hone their skills to work efficiently in this way, but the rewards can be huge for distributed teams.

Appendix: Mailing lists at the ASF

At the ASF we use mailing lists as our central asynchronous communications channel, based on our if it didn't happen on the dev mailing list, it didn't happen rule [2]. Mailing lists might be seen as tools of the past when you compare them with the latest shiny tools, but they remain a ubiquitous way of communicating in loosely coupled remote groups, especially when used with a strong discipline of Precise Quoting [3] and paying attention to meaningful subject lines. Unfortunately I hear some "modern" email clients make a mess of that quoting - just stay away from them.

References

[1] http://www.paulgraham.com/makersschedule.html - Paul Graham, Maker's Schedule, Manager's Schedule, July 2009

[2] https://community.apache.org/apache-way/apache-project-maturity-model.html - The Apache Project Maturity Model, ASF community development team, 2015.

[3] http://s.apache.org/gianugo_quoting_2002 - Gianugo Rabellino "[OT/Rant] Quoting", message to the cocoon-dev mailing list, January 2002

[4] http://www.apache.org/foundation/voting.html - ASF voting rules, created in 1999 probably, or even earlier among the Apache Group.

# # #

"Success at Apache" is a new monthly blog series that focuses on the processes behind why The Apache Software Foundation (ASF) "just works". First article: Project Independence https://s.apache.org/CE0V Second article: "All Carrot and No Stick" https://s.apache.org/ykoG

Posted at 12:42PM Feb 06, 2017 by Sally in SuccessAtApache | |

Monday January 09, 2017

Success at Apache: "All Carrot and No Stick"

By Danny Angus

When the ASF launched their "Success at Apache" series I offered to share my own experiences. If you read on, remember that this is my personal experience and that others may disagree with me, but as you'll see, that's really part of the fun.

For a bit background I’m currently the Project Management Committee (PMC) Chair of Apache Labs and in my day job I’m a "Divisional CTO" for a FTSE250 technology company. I first came to the ASF around 2000 when I was part of a startup - I was a CTO then too, it was the dot com boom, and it was just me and a couple of guys. We were considering a partnership with some researchers who wanted to commercialise their work, and were looking for a bit of software that we could use as the foundation for a product because a) we couldn’t afford to write it or buy it, and b) we didn’t have the knowledge anyway. What I found was Apache James http://james.apache.org , so I downloaded it, got it up and running, and did some prototyping, but we quickly realised that it needed work if we were going to be able to use it in production. I dug into it a little, subscribed to the mailing lists, asked questions and figured out what needed to be done to fix and extend what was already there, then started to modify it locally. Meantime I found myself answering other users’ questions on the user list, and one day noticed that I was actually answering more questions that I was asking. Shortly after that, that I was answering more questions than anyone else. Then I started submitting patches to the developer list (this was in the days of CVS: long before git!), which were reviewed and committed for me by the committers … but eventually they got bored with that and decided to extend commit privileges to me so I could do it all myself.

My experience illustrates an important characteristic of Apache projects: the fact that you can just turn up and get involved. Another very other important characteristic is that we are a meritocracy: demonstrating your capability is all you need to do in order to gain more responsibility; demonstrating your willingness and trustworthiness should be enough to get you the job. "Karma" is a word that is used to mean "access permission" in many Open Source projects, and we used to say that if you knew how to ask for karma properly, that was itself a sign that you could be trusted with it. Of course we were a much different organisation in those days, but the principles of a community built on merit and trust are still core to our identity. It's no coincidence that organisations cannot be part of our community: only individuals. Organisations are an important part of the world in which we exist, but we don't exist for their benefit, we only exist at all because as individuals we each bother to turn up and do stuff, from the guy who one time downloads and installs the Apache HTTP Web Server to Sam Ruby, our current (and can I just say excellent) President, everyone is contributing in their own way to the life of Apache and achieving benefits suited to their own, personal, motivations. So it was OK for me to focus on my own and my employer's priorities, which meant that I could learn from my new friends, develop the software we needed at work and become part of this amazing community all at the same time.

My experience of Apache is that it is what I would call "all carrot and no stick". I think that is the most healthy model of Open Source, as it is predicated on the fact that every participant will benefit from their participation without the need to contribute more than they are prepared to do. For me, focusing my contribution on the things I knew about was not only the most efficient use of my time, in terms of meeting our company's product goals, but it also allowed me to learn from others who had, and continue to have, way more knowledge and experience than I, and to benefit from their work. Mixing with these amazing people, many of whom are now real friends of mine, has taught me more than I would ever have learned any other way.

At this point in my involvement Apache went through a bit of what has diplomatically been described as "navel gazing", and settled on the idea that the organisational structure should be very very flat, and there should be no limit to our growth. As long as our standards were met by projects and people, we would welcome them both into our community. Those standards are partly about merit, partly about legal protection, one of the key roles Apache plays is to provide a degree of protection to projects and the people contributing to them, from the threat of bullies, trolls, and gorillas with expensive lawyers; and partly about ensuring that the behaviours and practices that define our identity and have contributed to the survival and the success of our organisation are continued by new generations of people in new projects using and creating technologies that we could hardly have dreamed about 16 years ago.

Before long the dust settled and I found myself voted to chair the Apache James Project http://apache.org/foundation/governance/pmcs.html , which was a whole new dimension of interesting. Chairing a project using only positive motivation teaches you a lot about people, including yourself, and I have a few observations about successful collaboration that I have found to be helpful both at work, where I strive to implement bottom-up decision making, and at the ASF where I want to make a positive contribution and see our communities flourish:

Free your mind.The collective sense of direction may not be what you expect, there have been times when I have been very sceptical about the reality of great sounding ideas, but I have also learned that it’s OK to go down the wrong road because most of the time it makes little difference in the end, usually you learn a lot regardless, and if people are really behind it you stand a much better chance of success than if the really good idea has all the fun of a death march. One phrase which is often used to summarise the spirit of Apache is “Community over code”, put the community first, and the code will follow.
Listen, and be supportive. There are a lot of different people involved in our projects with a lot of very different motivations. They are mostly all valid, and mostly all equally important if that even has an absolute scale. There are students studying our code, asking questions using our software and maybe fixing defects so that they can learn, there are employees of corporations who are being paid to protect their investment, to implement the product roadmap and maintain some predictable velocity, there are researchers who are pushing the boundaries of their chosen topic, there are people whose livelihood and success depends on a project, and those who are involved because it is a release from the pressure of things with names like "impact", "benefits", "deadlines" and "goals". Moderate or steer the discussion to ensure that all sides are heard, a meritocracy needs to listen to everyone not just the most vocal or assertive, and when I say listen that doesn’t mean formulating your own response while someone else is talking. Support people who you agree with, help to realise other people’s ideas, collaboration is only achieved by being truly committed to each other’s success, not just your own.
"A's hire A's B's hire C's". Find, support, and mentor the next generation, when your success depends upon the community it makes sense for you to put some effort into creating the best community you can.
Use Positive Language. When I was a kid being mean to my sister, adults used to say, "If you don't have anything nice to say, don't say anything at all". That's great advice if you’re involved in any collaborative venture, but doubly so when it is something like an Open Source project where you are usually communicating using written English, with people you don't know well, who might not have the same language skills as you do, who live in a different time zone and sometimes have very different cultural background than you. On top of all that you"re often debating the details of highly abstract technical concepts. The communication barrier itself can cause a kind of baseline of frustration so go easy on the negativity, one thing I like to do when I strongly disagree with someone is to write how I feel, then try to reword it using only positive language, it might sound like touchy-feely hippy nonsense to you, but you will be surprised how effective changing "I think you’re wrong and here’s why..." into "You have clearly thought a lot about this, I wonder if you have considered...". Alienating people is not the way to get your point across.
Learn to be a good loser. You don't own your projects, not here, and you're not the smartest person here either (OK so that’s not going to be 100% true, but there are 5,938 Committers today which makes it about 99.98%) recognising that and learning to embrace the collective view is hard for some people, but being able to step outside your subjective point of view and make a success of something you didn't believe in is a lesson in leadership that is definitely worth learning, because if not, your growth will be limited by the ideas that come from your own head, not accelerated by other people.
We are making it up as we go along http://apache.org/foundation/how-it-works.html . Yes, it sometimes seems from the outside like we have it all sorted and nailed down, and that we want to lawyer up and suck the life out of every fun thing (I mean we have a major software licence with our name on it, how grown up is that for goodness sakes?) But the truth is that Apache, The Apache Software Foundation is, and probably always will be, a work in progress, hopefully will be at-least-good-enough to survive the next unexpected storm, and there have been several of those already, but the only way we ever find that out is when it hits us. Over a relatively long period we have figured out, adopted, borrowed, adapted, had donations of, and thunk out with nothing but our own brains, a whole load of ideas about effective Open Source collaboration, governance, legal shenanigans, marketing, community building, and so on. Things that work well, some that mostly work, and some that are sometimes rubbish, but better than nothing. We write these things down and propagate this good practice amongst projects because it is the bedrock on which our foundation rests, but that doesn’t mean that it can’t change, we correct, adapt and evolve our best practices all the time, this is how we adapt, this is how we have survived and remained relevant in a field that seems to change almost beyond recognition every four or five years. And, being a meritocracy, if you don’t agree with the way things are, if you think it is out of date or ineffective or pointless, don’t complain, stay and fix it. We have another saying which is that "you can scratch your own itch" - don’t be passive, if you care about it, do it.

The important point about Apache is not that we have rules and committees but that we have these things because they have been shown to help us do the right thing, to help us to live by our principles and to provide a home for Open Source projects that will equip them to survive amongst the commercial sharks in the ocean of the software industry.
Finally: Define your own achievements. Whether you are doing it because you need some software, or because, like me, you just found it and it wasn't quite ready, whether you want to make friends, or to learn something new, whether it is because you are being paid to promote your employer's best interest, because you want to explore new ideas, or because you always wanted to write a book, Success at Apache is yours to define. Create your own measure of success and let us achieve it together.

# # #

Posted at 10:59AM Jan 09, 2017 by Sally in SuccessAtApache | |

Monday December 05, 2016

Success at Apache: Project Independence

By Mark Thomas

I've been involved in The Apache Software Foundation (ASF) since 2003. I was using Apache Tomcat at work and I hit a problem that needed a new feature to be implemented. There was already an enhancement request in Bugzilla so I submitted a patch. After some re-work by the project committers, the patch was applied and the feature available in the next release. I enjoy problem solving, so I started to take a look at the other open Tomcat bug reports and my involvement grew from there to include Apache Commons, the Infrastructure Team, the Security Team and, most recently, the Board of Directors to which I was elected in March 2016.

Apache Tomcat has always been at the heart of my involvement and is where I spend most of my time. Tomcat started with a donation to the ASF by Sun in 1999 and, some seven major versions later, the project continues to be very successful. A significant part that success is due to the involvement of a wide range of individuals from different companies. The reason those companies are happy co-operating on Tomcat is because of the importance the ASF places on project independence.

There are many aspects to project independence but, for me, the most important is that committers and Project Management Committee (PMC) members contribute to the project as individuals and do so with the intention of doing what is best for the community as a whole. Some committers contribute in their free time – I did for the first five years or so with Tomcat – and some are allowed /directed to spend time contributing to Apache projects by their employer. However, those committers contributing on their employer's time still need to act in the best interests of the community rather than the best interest of their employer.

To give a specific example, my employer has a product that is built around Apache Tomcat. The sales folks at my employer asked if I could add a feature to this product. The problem was that this feature required access to low-level Tomcat internals in order to implement it effectively. For this to be possible, I would have needed to make some ugly API changes to Tomcat to provide the integration points required. Rather than try and push those changes through, I persuaded my employer that it would better to donate the entire feature to the Apache Tomcat project.

This feature also demonstrates other important elements of a successful ASF project: the ability to make decisions in public and always aiming to achieve community consensus with those decisions. As the development of this new feature progressed, the design evolved as the community reviewed the commits and suggested improvements. This isn't always the quickest way of working but the quality of the end result – both technically but more importantly in terms of community health - more than makes up for that.

The perception of project independence is as important as projects actually being independent. It is a key factor in many projects choosing the ASF as their home so projects need to ensure that the perception agrees with reality.

Things can and do go wrong. With 350 projects it is pretty much a given that there will be a handful of ongoing issues at any given time. For example, there might be an attempt to push a project in a particular direction or to suggest that some external entity controls / leads / manages the project. Typically these are self-corrected by the PMC. Sometimes the PMC needs help to resolve the issue e.g. from V.P. Brand Management or possibly the ASF Board.

Being a board member is often viewed as more significant than it is. I have no more status in Apache Tomcat, Apache Commons or any other project as a board member than I did before my election to the board. I can still have bad ideas and my fellow community members still point it out when it happens. I don't get to always have my way just because I am board member. It is the board as a whole, rather than the individual board members, whose voice carries significant weight. It is fairly rare for any board member to speak on behalf of the board. To give that some context, I've probably done it no more than once a month since joining the board. It is sufficiently rare that board members always include an explicit "on behalf of the board" when speaking for the board rather than as an individual. Sometimes this point isn't appreciated and the views of an individual board member are incorrectly taken to be the views of the board.

The ASF board is also very different to a corporate board. The board manages the Foundation but it is the PMC that manages the project and sets the direction. The board has no role in the technical direction of a project. The board has responsibility for corporate governance, finance, legal etc., but its primary role is monitoring, mentoring and coaching our project communities to help keep them healthy. As part of this, the board reviews all projects on a regular basis. Newly graduated projects are reviewed monthly for typically 3 months before moving to quarterly reviews. The project V.P. (PMC Chair) is an important part of this. They are the eyes and ears of the board. While the board will look for warning signs as part of its regular review, the V.P. has much more in depth knowledge of the project and can flag specific issues early. Where issues are identified, the aim is to get the PMC to self-correct. The board will provide mentoring / coaching / guidance as necessary but it will be the PMC members who do the work to correct the issue.

As an example of the board working with a PMC, earlier this year the V.P. for a particular project became unavailable. The board became concerned because the regular reports were not being produced for the project. In this instance, no-else on the PMC had experience of being a project V.P so the board worked with the PMC to identify a new V.P. and to then mentor the new V.P. as they found their way in their new role.

For the last 17 years, the ASF has provided a home for a large and diverse set of open source projects. Key to this success has been the importance the ASF places on project independence as part of the Apache Way. By continuing to adhere to the principles of the Apache Way, I am confident that the ASF will continue to be successful for another 17 years and a long way beyond.

Posted at 12:59PM Dec 05, 2016 by Sally in SuccessAtApache | |

The Apache Software Foundation
Blogging in Action.

The Apache Software Foundation Blog

Semi-asynchronous decision making at the ASF

Try it yourself!

Appendix: Mailing lists at the ASF

References

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation

Sun	Mon	Tue	Wed	Thu	Fri	Sat
« March 2018
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Today

The Apache Software Foundation Blogging in Action.

The Apache Software Foundation Blog

Semi-asynchronous decision making at the ASF

Try it yourself!

Appendix: Mailing lists at the ASF

References

Calendar

Search

Hot Blogs (today's hits)

Tag Cloud

Categories

Feeds

Links

Navigation

The Apache Software Foundation
Blogging in Action.