Archive of Our Own

Every month or so the OTW will be doing a Q&A with one of its volunteers about their experiences in the organization. The posts express each volunteer's personal views and do not necessarily reflect the views of the OTW or constitute OTW policy. Today's post is with Frost the Fox, who volunteers on our Systems Committee.

How does what you do as a volunteer fit into what the OTW does?

As a volunteer for the Systems committee, we’re responsible for all of the OTW’s IT infrastructure, including our physical equipment such as our firewalls/routers, switches, and servers, as well as the many different software components that make up the Archive of Our Own, Fanlore, and our internal services that our committees use to conduct their business. Just like business IT departments, we provide and maintain the technical infrastructure to allow other committees to achieve their goals, and thus, the OTW’s goals.

What is a typical week like for you as a volunteer?

A typical week for Systems starts with our weekly meeting where we share our progress on any committee work we’re doing, bring up any important tasks that need to be done, and just generally catch up. Throughout the week we usually receive at least a couple of tickets for updates to some of our internal tools, which are usually pretty quick to take care of. Beyond this, things are different every week depending on what arises!

Sometimes we’ll have projects, requests from committees, or software updates that require changes to our configuration. In this case, we’ll use our configuration management system to implement the changes, and have them reviewed by at least 1 other person in the committee for sanity before deploying them. And of course, all infrastructure has its problems here and there. When there’s an outage on the AO3 or any of our other projects/tools, we respond and investigate the cause of the problem. We’ll identify the problem and liaise with other committees as needed to work towards a solution, and once we have one, implement it and monitor to ensure that things recover as expected.

What made you decide to volunteer?

I have a number of years of experience in the IT field ranging from generic help desk type work, to programming, to system administration. Having already been a part of the organization, I knew that there was a lot of infrastructure to maintain, and that there was not a huge amount of people to do so. I have a lot of fun working with technology, and volunteering with the OTW elsewhere had been very rewarding for me, so when I saw that Systems was recruiting, I knew I would be doing a disservice to myself if I didn’t apply.

What has been your biggest challenge doing work for the OTW?

My work in my career in IT has been mostly focused towards general business operations which is focused a lot on internal services, as opposed to the OTW, which has more public services. While the basics of server operating systems, virtualization and the like transferred over, there were a lot of other technologies I wasn’t so familiar with that I had to learn, particularly those surrounding the Archive. Getting used to any organization and how their IT systems connect also takes some time. I wasn’t familiar with the configuration management software in use, so that in and of itself was a major learning curve for me, but once I became familiar with it, it made understanding what specific machines do so much easier.

There’s also the nervousness that many people have felt when deploying something to their production environment, except here it’s on a much larger scale than I’d ever dealt with before!

What fannish things do you like to do?

Recently I’ve gotten back into a bit of a writing mood, and I’ve been doing some work on an original story I played around with over a year ago. While that’s not something I’m likely to put on the Archive, it’s gotten the creative juices flowing, and I have some potentially fun ideas for some of my fandoms that I’ll be exploring! I mostly read fic so it’ll definitely be a new experience for me to put something out there for others to enjoy.

Speaking of reading, I’ve been doing a lot of that whenever my schedule permits. Some of that has been in my original fandoms such as Black Lagoon, Final Fantasy, Red Rising & Zootopia, and some in a couple of new fandoms like Hitman. I’m also working on finishing a couple of series and playing some different games that might also lead to new fandoms for me later on. In general I’m always bouncing around between different fandoms looking for something fun to read!

Now that our volunteer’s said five things about what they do, it’s your turn to ask one more thing! Feel free to ask about their work in the comments. Or if you'd like, you can check out earlier Five Things posts.

The Organization for Transformative Works is the non-profit parent organization of multiple projects including Archive of Our Own, Fanlore, Open Doors, Transformative Works and Cultures, and OTW Legal Advocacy. We are a fan run, entirely donor-supported organization staffed by volunteers. Find out more about us on our website.

Every month or so the OTW will be doing a Q&A with one of its volunteers about their experiences in the organization. The posts express each volunteer's personal views and do not necessarily reflect the views of the OTW or constitute OTW policy. Today's post is with Matthew Vernon, who volunteers as chair of our Systems Committee.

How does what you do as a volunteer fit into what the OTW does?
I am the Chair of Systems, which is the committee responsible for managing hardware and IT infrastructure for the OTW as a whole (not just AO3!) We work closely with a number of other OTW committees, particularly AD&T who manage the software design and development of AO3.

As Chair I do a range of things -- I run our weekly meetings, manage our volunteers, liaise with the OTW Board and Chairs of other committees, keep an eye on our ticket queue, and do quite a lot of code review.

What is a typical week like for you as a volunteer?

The one constant is our weekly meeting (on a Sunday evening in UK time), when we catch up as a team, talk about where we're up to and plan the week ahead. Beyond that, it depends a bit on what needs doing, and how much free time I have! I review some merge requests for our configuration management system almost every week, and correspondence with some other part of the OTW is also a regular feature.

What made you decide to volunteer?

I became aware of the OTW through Yuletide, the annual rare fandoms gift exchange. Some friends of mine were running writing parties, and it seemed like fun! That introduced me to AO3. When OTW advertised for some sysadmins, it seemed like an obvious way to give something back to the OTW, since I'm a sysadmin in my day job.

What has been your biggest challenge doing work for Systems?

Systems do a lot of work with not a lot of people-power. That's really good, but it means there is also often quite a lot going on, and it can sometimes be hard to keep up with the important but less immediately urgent tasks. Being Chair means I don't do much direct technical work myself these days, too!

What fannish things do you like to do?

Covid-19 lockdown has given me more time at home, so I've been re-watching some of my favourite shows. I'm also really looking forward to this year's Yuletide, and the joy people get from my distinctly average writing :)

Now that our volunteer’s said five things about what they do, it’s your turn to ask one more thing! Feel free to ask about their work in comments. Or if you'd like, you can check out earlier Five Things posts.

The Organization for Transformative Works is the non-profit parent organization of multiple projects including Archive of Our Own, Fanlore, Open Doors, Transformative Works and Cultures, and OTW Legal Advocacy. We are a fan run, entirely donor-supported organization staffed by volunteers. Find out more about us on our website.

When asked to write up a few words about his time with the OTW and AO3 in particular, james_ had so much to say that he had trouble sticking to the word count. (He’d like to thank Priscilla for helping him to edit!) Below you can read about some of the tougher times that james_ has seen during his time as a member of the Systems and Accessibility, Design and Technology committees. You can also hear about the rewards he’s gained from his hard work to keep our vision clear and our morale high. As you can see below, james_ was amongst the staffers who accepted the Hugo Award for Best Related Work on behalf of AO3.

Volunteering for the OTW in the early days was exciting, stressful, exhausting, and demoralising, but also worth it. At that time we were working with five servers and we were constantly adjusting the load between the few systems we had. We reached out to our friends at Dreamwidth (thanks, Mark) and they helped us. We were learning even as the tsunami of growing AO3 traffic beat down upon our shore.

While there are always people willing to try and pull you down, they are greatly outnumbered by those supporting us and buoying us up. I am grateful to each person who donates to the OTW. Your donations mean that we can afford the machines that keep the Archive running stably, and that nowadays I rarely get woken in the middle of the night due to unexpected downtime.

Something else that has had a significant impact in my volunteering life were the recurring conflicts both my committees had with previous iterations of the OTW Board of Directors. These were a source of great frustration and I even contacted the Legal committee to see how OTW members could call the board to account. After the resignation of the entire 2015 board, things have been much better. No organization is perfect, but I believe everyone in the OTW is very much happier today. I hope this will continue and believe the best way to do that is to ensure that every election is properly contested; I stood for election myself in 2016 and would do so again if necessary to make sure that there were enough candidates.

Our successes have been external as well as internal. This year, I had the pleasure of standing on the stage at Worldcon as AO3 won a Hugo Award and it was such a joy.

As for the future, I believe that we will need to raise significantly more than we do today in order to hire paid employees. We cannot sustainably run forever on purely volunteer labor. We get roughly 5% of Wikipedia's pageviews and our budget is about one-third of one percent of theirs.

The Archive of Our Own started out as a pipe dream: What if we, the fans, actually owned the servers that house our fanworks? What if we got to write our own Terms of Service, and weren't dependent on for-profit companies to host our fannish spaces?

Defying the odds, the Archive of Our Own settled into its first set of servers in September 2009. A few months later, the initial framework for managing accounts, posting fics, and exchanging comments was considered ready for public testing: The AO3 entered its "open beta" phase, allowing anyone to sign up for an invitation and create an account.

In the last five years and change, we've seen the Archive grow in a way nobody even dreamed of when those servers were put in place. Today, we host over 1.5 million fanworks and count 7 million visitors every month. To accommodate this growth, we've been adding servers and improving our infrastructure along the way.

Three weeks ago, we reached an exciting new milestone in our history. The servers owned by the OTW moved into their very own rack (as opposed to each sharing a space with other servers across the colocation facility).

Now that our servers have a rack of their own, our tireless Systems volunteers are hard at work planning new server purchases to replace some of these old machines. All this has only been possible thanks to your generous donations to the OTW, and from the bottom of our excited hearts, we thank you!

The OTW's Systems teams work behind the scenes to support, manage, and maintain all the technical systems needed to run the OTW and its projects, such as the Archive of Our Own and Fanlore.

Systems' work mostly happens behind the scenes, but they are BUSY, fielding requests from all parts of the organization and working hard to keep all our sites up and responsive. Systems team members have to be 'on call' in order to deal with emergencies at any time of the day or night: if the Archive of Our Own goes down, it's Systems who fly to the rescue (while over 130 thousand users wait impatiently!).

2012 was a particularly demanding year for Systems because of the speed with which the OTW and its projects grew. Over 2,970,103 people now access the Archive of Our Own in the course of a month, up from 808,000 a year ago. Meanwhile, Fanlore has also grown, passing 400,000 edits in 2012, and other projects have continued to develop. Managing these projects and their volunteers also requires technical resources, and Systems have helped the OTW to transition to some more effective tools over the past year.

Systems highlights

Over the course of 2012, Systems:

• Handled 557 requests from around the organization \0/
• Transitioned the OTW website and some related tools and projects to a new host with a third party Drupal vendor, who will provide much-needed technical support for these tools.
• Dealt with the performance problems on the Archive of Our Own, stepping in to implement major performance enhancements and keep the site up.
• Researched, bought and installed 3 new servers to host our projects and cope with the ever-growing demands on the Archive of Our Own.
• Researched hosting options and installed two additional servers after a kindly benefactor donated them to the OTW.
• Set up new hosting and tools for our volunteers to use, including new hosted environments for our coders, so that coders don't have to install the Archive code on their own machines.
• Kept everything up and running, with amazing patience and good humour in the most stressful situations.

Find out more!

James from Systems has written up an amazing and detailed account of the main work Systems did in the course of 2012. To get some in-depth insight into the amazing work Systems do, check out: A year with the Systems team

If you're technically minded, or curious about how much hardware is needed to run the Archive of our Own, you'll also enjoy James' posts on our changing server setups over the past year, and our technical plans going forward:

January 2012 server setup
January 2013 server setup
Going forward: our hardware setup and technical plans

Thank you!

Systems do an amazing job of juggling their many responsibilities. We really appreciate their work - thanks Systems!

The OTW's Systems teams work behind the scenes to support, manage, and maintain all the technical systems needed to run the OTW and its projects, such as the Archive of Our Own and Fanlore.

Systems' work mostly happens behind the scenes, but they are BUSY, fielding requests from all parts of the organization and working hard to keep all our sites up and responsive. Systems team members have to be 'on call' in order to deal with emergencies at any time of the day or night: if the Archive of Our Own goes down, it's Systems who fly to the rescue (while over 130 thousand users wait impatiently!).

2012 was a particularly demanding year for Systems because of the speed with which the OTW and its projects grew. Over 2,970,103 people now access the Archive of Our Own in the course of a month, up from 808,000 a year ago. Meanwhile, Fanlore has also grown, passing 400,000 edits in 2012, and other projects have continued to develop. Managing these projects and their volunteers also requires technical resources, and Systems have helped the OTW to transition to some more effective tools over the past year.

Systems highlights

Over the course of 2012, Systems:

• Handled 557 requests from around the organization \0/
• Transitioned the OTW website and some related tools and projects to a new host with a third party Drupal vendor, who will provide much-needed technical support for these tools.
• Dealt with the performance problems on the Archive of Our Own, stepping in to implement major performance enhancements and keep the site up.
• Researched, bought and installed 3 new servers to host our projects and cope with the ever-growing demands on the Archive of Our Own.
• Researched hosting options and installed two additional servers after a kindly benefactor donated them to the OTW.
• Set up new hosting and tools for our volunteers to use, including new hosted environments for our coders, so that coders don't have to install the Archive code on their own machines.
• Kept everything up and running, with amazing patience and good humour in the most stressful situations.

Find out more!

James from Systems has written up an amazing and detailed account of the main work Systems did in the course of 2012. To get some in-depth insight into the amazing work Systems do, check out: A year with the Systems team

If you're technically minded, or curious about how much hardware is needed to run the Archive of our Own, you'll also enjoy James' posts on our changing server setups over the past year, and our technical plans going forward:

January 2012 server setup
January 2013 server setup
Going forward: our hardware setup and technical plans

Thank you!

Systems do an amazing job of juggling their many responsibilities. We really appreciate their work - thanks Systems!

The OTW's Systems teams work behind the scenes to support, manage, and maintain all the technical systems needed to run the OTW and its projects, such as the Archive of Our Own and Fanlore.

Systems' work mostly happens behind the scenes, but they are BUSY, fielding requests from all parts of the organization and working hard to keep all our sites up and responsive. Systems team members have to be 'on call' in order to deal with emergencies at any time of the day or night: if the Archive of Our Own goes down, it's Systems who fly to the rescue (while over 130 thousand users wait impatiently!).

2012 was a particularly demanding year for Systems because of the speed with which the OTW and its projects grew. Over 2,970,103 people now access the Archive of Our Own in the course of a month, up from 808,000 a year ago. Meanwhile, Fanlore has also grown, passing 400,000 edits in 2012, and other projects have continued to develop. Managing these projects and their volunteers also requires technical resources, and Systems have helped the OTW to transition to some more effective tools over the past year.

Systems highlights

Over the course of 2012, Systems:

• Handled 557 requests from around the organization \0/
• Transitioned the OTW website and some related tools and projects to a new host with a third party Drupal vendor, who will provide much-needed technical support for these tools.
• Dealt with the performance problems on the Archive of Our Own, stepping in to implement major performance enhancements and keep the site up.
• Researched, bought and installed 3 new servers to host our projects and cope with the ever-growing demands on the Archive of Our Own.
• Researched hosting options and installed two additional servers after a kindly benefactor donated them to the OTW.
• Set up new hosting and tools for our volunteers to use, including new hosted environments for our coders, so that coders don't have to install the Archive code on their own machines.
• Kept everything up and running, with amazing patience and good humour in the most stressful situations.

Find out more!

James from Systems has written up an amazing and detailed account of the main work Systems did in the course of 2012. To get some in-depth insight into the amazing work Systems do, check out: A year with the Systems team

If you're technically minded, or curious about how much hardware is needed to run the Archive of our Own, you'll also enjoy James' posts on our changing server setups over the past year, and our technical plans going forward:

January 2012 server setup
January 2013 server setup
Going forward: our hardware setup and technical plans

Thank you!

Systems do an amazing job of juggling their many responsibilities. We really appreciate their work - thanks Systems!

The Systems team is responsible for all the ‘behind the scenes’ parts of the OTW’s technical work; in particular, we maintain the servers which run all the OTW’s projects, including the Archive of Our Own and Fanlore. As the OTW has grown, so has our job, and we’ve been very busy over the past twelve months!

This update gives an overview of some of the key changes we’ve made this year. While it doesn’t cover every detail, we hope it will give our users a sense of the work we’ve done and (importantly) where we’ve spent money. We’ve included quite a few technical details for those users who are curious, but hope that non-technical users will be able to get the gist.

January 2012

At the start of January 2012, we were maintaining 12 servers: 6 physical machines and 6 virtual ones. You can see more details in January 2012 - our server setup.

February

The Archive of Our Own was suffering performance problems as more users joined the site. We spent time working to make things more reliable and balancing unicorns. We had to disable our online web tracking system (piwik), as it caused slow responses with the Archive. Although our work helped performance, server OTW2 (running Archive-related services) started collapsing under the load.

March

We implemented a system which killed off runaway processes that were created when users were downloading works from the Archive of Our Own.

April

A bug caused Linux systems to have performance issues when its uptime reached 200 days. As our servers all run Linux, we were affected. A new kernel and a reboot on our Linux-based servers fixed the problem very quickly \0/.

June - a month of many happenings!

Our long-serving staffer Sidra stepped down as Technical lead and joint Chair of the Systems group. We have missed her and hope to see her rejoin us in the future.

In response to the rising numbers of visitors to the AO3, we upgraded our colocation bandwidth (the amount of network traffic) to an unmetered 100Megabits/second, which cost an additional $100 per month.

Demands on our servers were also increasing behind the scenes, as the number of coders and the complexity of the Archive meant that the webdevs (used by our coders to develop new code) and the Test Archive, where we test out new code before releasing it onto the live site, were unusable. We upgraded the servers these were hosted on, which increased our virtual server bill by an additional $200 per month.

We decided that we had reached a size where it would be worth buying our own servers rather than using virtual servers for the webdevs. We investigated the costs of buying new servers, but happily later in the month, two servers were donated to OTW. We then started the long task of finding a suitable hosting provider, as the servers were located a long way from our main colocation host and shipping costs were high.

Performance issues on the Archive of Our Own were at their height during June, and we spent lots of time working to address these issues. Some parts of the site were unable to cope with the number of users who were now accessing the site: in particular, we had significant problems with server OTW5 and the demands created by the tag filters, which required a lot of space for temporary files.

In order to reduce the demands on the servers, we implemented Squid caching on the Archive, which alleviated some of the problems. On the 13th of June we decided to disable the tag filters and the Archive became significantly more stable. This reduced the amount of hour by hour hand holding the servers needed, giving our teams more time to work on longer-term solutions, including the code for the new version of the filters.

July

The first of July brought a leap second which caused servers around the globe to slow down. We fixed the issue by patching the servers as needed and then rebooting - with just half an hour turnaround!

We consulted with Mark from Dreamwidth about the systems architecture of the Archive. We got a couple of very useful pointers (thanks, Mark!) as well as concrete advice, such as increasing the amount of memory available for our memcache caching.

A disk died in server OTW2 and a replacement disk was donated by a member of the Systems group.

We started to use a large cloudhosted server space to develop the new system that would replace the old tagging system. This machine was not turned on at all times, only when the developers were coding, or when the new system was being tested. Hiring this server space allowed us to develop the code on a full copy of the Archive’s data and do more effective testing, which more closely replicated the conditions of the real site. Since the filters are such an important part of the AO3, and have such big performance implications, this was very important.

We upgraded the RAM on servers OTW3, OTW4 and OTW5. We replaced all of the RAM in OTW5 and put its old RAM in OTW3 and OTW4. This cost approximately $2,200 and gave us some noticeable performance improvements on the Archive.

We also upgraded the main webserver and database software stack on the Archive.

And lastly, it was SysAdmin Day. There was cake. \0/

August

We started using a managed firewall at our main colocation facility. This provides both a much simpler configuration of the main network connection to the servers, and allows secure remote access for systems administrators and senior coders. It costs an additional $50 per month.

A typo in our DNS while switching over to this allowed a spammer to redirect some of our traffic to their site. Happily we were able to fix this as soon as the problem was reported, although the fix took a while to show for all users. The firewall changes also caused a few lingering issues for users connecting via certain methods; these took a little while to fix.

September

We purchased battery backup devices for the RAID controllers on OTW1 and OTW2, meaning their disk systems are much more performant and reliable. The batteries and installation cost a total of $250.

A hardware based firewall (Mikrotik RB1100AHx2) was purchased and configured for the new colocation facility, costing around $600.

Systems supported the coders in getting the new embedded media player to work on the Archive.

We also migrated transformativeworks.org, Elections and Open Doors to a third party Drupal supplier.

October

The donated, dedicated hardware for Dev and Stage (our webdev and test servers) were installed in their new colocation site, after long and hard hours spent investigating options for hosting companies and insurance. After installation the initial configuration required to run the Archive code was completed. These machines support a larger number of coders than was previously possible, giving them access to a hosted development environment to run the Archive. The hosting cost is approximately $400 per month.

We were able to decommission the virtual machine that was the Dev server (for webdevs) immediately, saving $319 per month - so the new hosted servers are only costing us about $80 more than the old setup. Considerable work was done to get Elastic Search working in our dev, test and production environments (production is the live Archive).

November

We were running out of disk space on OTW5, which is critical to the operation of the Archive. We purchased a pair of 200GB intel 710’s and adapters which were installed in OTW5, for a total cost of $1,700. These disks are expensive, however they are fast and are enterprise grade (meant for heavy production use) rather than home grade, which is significant on a site such as ours. Solid state drives (SSDs) are dependent on the amount of use they endure and the 710’s are rated at an endurance of 1.5PB with 20 percent over provisioning (meaning they will last us far longer than a home grade SSD).

At roughly the same time, the tag filters were returned to the Archive using Elastic Search. There was much rejoicing.

December

We were waiting until the tag filters were back in place before deciding what servers we would need to buy to provide the Archive with enough performance to continue to grow in the following year. After discussing budgets with Finance and Board, we put a proposal through for three servers for a total price of $28,200. We arrived at this price after checking with a number of vendors; we went for the cheapest vendor we were happy with. The difference in price between the cheapest and most expensive vendor was $2,600. The servers will be described in January 2013 - server setup.

Having bought the servers, we needed to host them. We had to decide whether to rent a whole 19-inch rack to ourselves or to try and and squeeze the servers into existing space in our shared facility. In the long term we will likely require a 19-inch rack to ourselves, but as this will cost about $2,100 per month we worked hard to find a way of splitting our servers into two sections so that we could fit them into existing space.

We did this by moving all the Archive-related functions from OTW1 and OTW2, then moving the machines and the QNAP to another location in the facility. At this point we discovered that the QNAP did not reboot cleanly and we had to have a KVM installed before we could get it working. We are renting a KVM (at $25 per month) until we can reduce the reliance on the QNAP to a minimum.

January and February 2013

So far in 2013, we’ve been working to set up the new servers. You can see the details of our new servers and their setup in January 2013 - server setup, and find out more about our plans in Going Forward: our server setup and plans.

In closing

These are only the major items: there are many pieces of work which are done on a regular basis by all the members of the team. The Systems team averages between 30 and 50 hours a week on the organization’s business. The majority of the team are professional systems administrators/IT professionals and have over 90 years of experience between us.

Systems are proud to support the OTW and its projects. We are all volunteers, but as you can see from the details here, providing the service is not free. Servers and hosting costs are expensive! We will never place advertising on the Archive or any of our other sites, so please do consider donating to the Organization for Transformative Works. Donating at least $10 will gain you membership to the OTW and allow you to vote in our elections. (Plus you will get warm fuzzies in your tummy and know you are doing good things for all of fandom-kind!)

The Systems team is responsible for all the ‘behind the scenes’ parts of the OTW’s technical work; in particular, we maintain the servers which run all the OTW’s projects, including the Archive of Our Own and Fanlore. As the OTW has grown, so has our job, and we’ve been very busy over the past twelve months!

This update gives an overview of some of the key changes we’ve made this year. While it doesn’t cover every detail, we hope it will give our users a sense of the work we’ve done and (importantly) where we’ve spent money. We’ve included quite a few technical details for those users who are curious, but hope that non-technical users will be able to get the gist.

January 2012

At the start of January 2012, we were maintaining 12 servers: 6 physical machines and 6 virtual ones. You can see more details in January 2012 - our server setup.

February

The Archive of Our Own was suffering performance problems as more users joined the site. We spent time working to make things more reliable and balancing unicorns. We had to disable our online web tracking system (piwik), as it caused slow responses with the Archive. Although our work helped performance, server OTW2 (running Archive-related services) started collapsing under the load.

March

We implemented a system which killed off runaway processes that were created when users were downloading works from the Archive of Our Own.

April

A bug caused Linux systems to have performance issues when its uptime reached 200 days. As our servers all run Linux, we were affected. A new kernel and a reboot on our Linux-based servers fixed the problem very quickly \0/.

June - a month of many happenings!

Our long-serving staffer Sidra stepped down as Technical lead and joint Chair of the Systems group. We have missed her and hope to see her rejoin us in the future.

In response to the rising numbers of visitors to the AO3, we upgraded our colocation bandwidth (the amount of network traffic) to an unmetered 100Megabits/second, which cost an additional $100 per month.

Demands on our servers were also increasing behind the scenes, as the number of coders and the complexity of the Archive meant that the webdevs (used by our coders to develop new code) and the Test Archive, where we test out new code before releasing it onto the live site, were unusable. We upgraded the servers these were hosted on, which increased our virtual server bill by an additional $200 per month.

We decided that we had reached a size where it would be worth buying our own servers rather than using virtual servers for the webdevs. We investigated the costs of buying new servers, but happily later in the month, two servers were donated to OTW. We then started the long task of finding a suitable hosting provider, as the servers were located a long way from our main colocation host and shipping costs were high.

Performance issues on the Archive of Our Own were at their height during June, and we spent lots of time working to address these issues. Some parts of the site were unable to cope with the number of users who were now accessing the site: in particular, we had significant problems with server OTW5 and the demands created by the tag filters, which required a lot of space for temporary files.

In order to reduce the demands on the servers, we implemented Squid caching on the Archive, which alleviated some of the problems. On the 13th of June we decided to disable the tag filters and the Archive became significantly more stable. This reduced the amount of hour by hour hand holding the servers needed, giving our teams more time to work on longer-term solutions, including the code for the new version of the filters.

July

The first of July brought a leap second which caused servers around the globe to slow down. We fixed the issue by patching the servers as needed and then rebooting - with just half an hour turnaround!

We consulted with Mark from Dreamwidth about the systems architecture of the Archive. We got a couple of very useful pointers (thanks, Mark!) as well as concrete advice, such as increasing the amount of memory available for our memcache caching.

A disk died in server OTW2 and a replacement disk was donated by a member of the Systems group.

We started to use a large cloudhosted server space to develop the new system that would replace the old tagging system. This machine was not turned on at all times, only when the developers were coding, or when the new system was being tested. Hiring this server space allowed us to develop the code on a full copy of the Archive’s data and do more effective testing, which more closely replicated the conditions of the real site. Since the filters are such an important part of the AO3, and have such big performance implications, this was very important.

We upgraded the RAM on servers OTW3, OTW4 and OTW5. We replaced all of the RAM in OTW5 and put its old RAM in OTW3 and OTW4. This cost approximately $2,200 and gave us some noticeable performance improvements on the Archive.

We also upgraded the main webserver and database software stack on the Archive.

And lastly, it was SysAdmin Day. There was cake. \0/

August

We started using a managed firewall at our main colocation facility. This provides both a much simpler configuration of the main network connection to the servers, and allows secure remote access for systems administrators and senior coders. It costs an additional $50 per month.

A typo in our DNS while switching over to this allowed a spammer to redirect some of our traffic to their site. Happily we were able to fix this as soon as the problem was reported, although the fix took a while to show for all users. The firewall changes also caused a few lingering issues for users connecting via certain methods; these took a little while to fix.

September

We purchased battery backup devices for the RAID controllers on OTW1 and OTW2, meaning their disk systems are much more performant and reliable. The batteries and installation cost a total of $250.

A hardware based firewall (Mikrotik RB1100AHx2) was purchased and configured for the new colocation facility, costing around $600.

Systems supported the coders in getting the new embedded media player to work on the Archive.

We also migrated transformativeworks.org, Elections and Open Doors to a third party Drupal supplier.

October

The donated, dedicated hardware for Dev and Stage (our webdev and test servers) were installed in their new colocation site, after long and hard hours spent investigating options for hosting companies and insurance. After installation the initial configuration required to run the Archive code was completed. These machines support a larger number of coders than was previously possible, giving them access to a hosted development environment to run the Archive. The hosting cost is approximately $400 per month.

We were able to decommission the virtual machine that was the Dev server (for webdevs) immediately, saving $319 per month - so the new hosted servers are only costing us about $80 more than the old setup. Considerable work was done to get Elastic Search working in our dev, test and production environments (production is the live Archive).

November

We were running out of disk space on OTW5, which is critical to the operation of the Archive. We purchased a pair of 200GB intel 710’s and adapters which were installed in OTW5, for a total cost of $1,700. These disks are expensive, however they are fast and are enterprise grade (meant for heavy production use) rather than home grade, which is significant on a site such as ours. Solid state drives (SSDs) are dependent on the amount of use they endure and the 710’s are rated at an endurance of 1.5PB with 20 percent over provisioning (meaning they will last us far longer than a home grade SSD).

At roughly the same time, the tag filters were returned to the Archive using Elastic Search. There was much rejoicing.

December

We were waiting until the tag filters were back in place before deciding what servers we would need to buy to provide the Archive with enough performance to continue to grow in the following year. After discussing budgets with Finance and Board, we put a proposal through for three servers for a total price of $28,200. We arrived at this price after checking with a number of vendors; we went for the cheapest vendor we were happy with. The difference in price between the cheapest and most expensive vendor was $2,600. The servers will be described in January 2013 - server setup.

Having bought the servers, we needed to host them. We had to decide whether to rent a whole 19-inch rack to ourselves or to try and and squeeze the servers into existing space in our shared facility. In the long term we will likely require a 19-inch rack to ourselves, but as this will cost about $2,100 per month we worked hard to find a way of splitting our servers into two sections so that we could fit them into existing space.

We did this by moving all the Archive-related functions from OTW1 and OTW2, then moving the machines and the QNAP to another location in the facility. At this point we discovered that the QNAP did not reboot cleanly and we had to have a KVM installed before we could get it working. We are renting a KVM (at $25 per month) until we can reduce the reliance on the QNAP to a minimum.

January and February 2013

So far in 2013, we’ve been working to set up the new servers. You can see the details of our new servers and their setup in January 2013 - server setup, and find out more about our plans in Going Forward: our server setup and plans.

In closing

These are only the major items: there are many pieces of work which are done on a regular basis by all the members of the team. The Systems team averages between 30 and 50 hours a week on the organization’s business. The majority of the team are professional systems administrators/IT professionals and have over 90 years of experience between us.

Systems are proud to support the OTW and its projects. We are all volunteers, but as you can see from the details here, providing the service is not free. Servers and hosting costs are expensive! We will never place advertising on the Archive or any of our other sites, so please do consider donating to the Organization for Transformative Works. Donating at least $10 will gain you membership to the OTW and allow you to vote in our elections. (Plus you will get warm fuzzies in your tummy and know you are doing good things for all of fandom-kind!)

By the beginning of January 2013, the OTW had acquired 5 new physical servers and Systems had begun the work of restructuring our server setup to use the new machines. Our aim was to use our existing hardware as effectively as possible, and to set things up so we can work more efficiently and add new hardware more easily in the future.

Short term work

Our current focus is on transitioning to the new machines. So far, we have completed the work to automate the installation of our database servers. After some testing, we moved the Archive database onto the new machines. The Archive’s unicorns (which serve up all the data) are now all running on the new machines, which means that the site is under much less pressure.

The next step is to move other services, including Squid (which takes care of some of our caching) and resque (which takes care of delayed jobs like sending out mail).

We plan to transition Fanlore, Transformative Works and Cultures (the OTW journal) and Symposium (TWC’s blog) on to their new physical server when the groups who use the service have completed their testing - this work will be done shortly.

Once all this work has been done and the old systems are no longer in production, we can rearrange the hardware to better support our requirements. The three machines OTW 3,4 and 5 are all the same chassis with different components. They are currently configured as follows:

OTW3: cpu 2*4 core @ 2.4GHz, 48GB of RAM, 4*SAS (147GB)
OTW4: cpu 2*4 core @ 2.4GHz, 48GB of RAM, 4*SAS (147GB)
OTW5: cpu 2*6 core @ 2.67GHz, 96GB of RAM, 2*SSD Intel X25 ( 80GB ) and 2*Intel 710 ( 200GB)

OTW5 was originally our database server, which is why we needed extra RAM. However, as it is now a secondary server (we have a shiny new database server) it will not be used efficiently. So, we’ll move the RAM from OTW5 to OTW3, along with the Intel 710 drives. OTW3 will become the new database slave. The new setup will be as follows):

ao3-db02 (was OTW3): cpu 2*4 core @ 2.4GHz, 96GB of RAM, 2*SAS (147GB) and 2*Intel 710 ( 200GB)
ao3-front01 (was OTW4): cpu 2*4 core @ 2.4GHz, 48GB of RAM, 2*SAS (147GB) and 2*SSD Intel X25 ( 80GB )
ao3-app03 (was OTW5): cpu 2*6 core @ 2.67GHz, 48GB of RAM, 4*SAS (147GB)

Once this is done, installing the new operating system and introducing them to the Archive should be relatively pain free. This will give us 8 machines running the Archive of Our Own, with their roles distributed in a much more efficient pattern.

Looking ahead

Systems have worked hard over the past year to improve the OTW’s hardware and use our resources as efficiently as possible. We’ll continue to work on maintenance; for example, we expect to upgrade the version of Debian on all our servers within the year (the new version will be released shortly). Upgrading our software means we have access to the latest security patches and our sites are as stable and secure as possible.

We’re also thinking past 2013. All our projects are growing, and with it will come increased demands on our infrastructure. If the Archive of Our Own continues to grow at its present rate, we’ll need to add more servers within a year. We also plan to add multimedia hosting in the future, which will demand even more server space and support. The Systems team is planning for this and thinking about the best ways to support expansion going forward. If you enjoy using the OTW’s projects, you can help support our expansion too by donating and reminding others that the OTW depends on your support!

By the beginning of January 2013, the OTW had acquired 5 new physical servers and Systems had begun the work of restructuring our server setup to use the new machines. Our aim was to use our existing hardware as effectively as possible, and to set things up so we can work more efficiently and add new hardware more easily in the future.

Short term work

Our current focus is on transitioning to the new machines. So far, we have completed the work to automate the installation of our database servers. After some testing, we moved the Archive database onto the new machines. The Archive’s unicorns (which serve up all the data) are now all running on the new machines, which means that the site is under much less pressure.

The next step is to move other services, including Squid (which takes care of some of our caching) and resque (which takes care of delayed jobs like sending out mail).

We plan to transition Fanlore, Transformative Works and Cultures (the OTW journal) and Symposium (TWC’s blog) on to their new physical server when the groups who use the service have completed their testing - this work will be done shortly.

Once all this work has been done and the old systems are no longer in production, we can rearrange the hardware to better support our requirements. The three machines OTW 3,4 and 5 are all the same chassis with different components. They are currently configured as follows:

OTW3: cpu 2*4 core @ 2.4GHz, 48GB of RAM, 4*SAS (147GB)
OTW4: cpu 2*4 core @ 2.4GHz, 48GB of RAM, 4*SAS (147GB)
OTW5: cpu 2*6 core @ 2.67GHz, 96GB of RAM, 2*SSD Intel X25 ( 80GB ) and 2*Intel 710 ( 200GB)

OTW5 was originally our database server, which is why we needed extra RAM. However, as it is now a secondary server (we have a shiny new database server) it will not be used efficiently. So, we’ll move the RAM from OTW5 to OTW3, along with the Intel 710 drives. OTW3 will become the new database slave. The new setup will be as follows):

ao3-db02 (was OTW3): cpu 2*4 core @ 2.4GHz, 96GB of RAM, 2*SAS (147GB) and 2*Intel 710 ( 200GB)
ao3-front01 (was OTW4): cpu 2*4 core @ 2.4GHz, 48GB of RAM, 2*SAS (147GB) and 2*SSD Intel X25 ( 80GB )
ao3-app03 (was OTW5): cpu 2*6 core @ 2.67GHz, 48GB of RAM, 4*SAS (147GB)

Once this is done, installing the new operating system and introducing them to the Archive should be relatively pain free. This will give us 8 machines running the Archive of Our Own, with their roles distributed in a much more efficient pattern.

Looking ahead

Systems have worked hard over the past year to improve the OTW’s hardware and use our resources as efficiently as possible. We’ll continue to work on maintenance; for example, we expect to upgrade the version of Debian on all our servers within the year (the new version will be released shortly). Upgrading our software means we have access to the latest security patches and our sites are as stable and secure as possible.

We’re also thinking past 2013. All our projects are growing, and with it will come increased demands on our infrastructure. If the Archive of Our Own continues to grow at its present rate, we’ll need to add more servers within a year. We also plan to add multimedia hosting in the future, which will demand even more server space and support. The Systems team is planning for this and thinking about the best ways to support expansion going forward. If you enjoy using the OTW’s projects, you can help support our expansion too by donating and reminding others that the OTW depends on your support!

January 2013 - our server setup

Over the course of 2012, the demands on all our sites grew. In particular, the number of users accessing the Archive of Our Own each month dramatically expanded from 963,818 in January 2012 to 2,970,103 by December 2012. This demanded a significant expansion of our hardware: we bought 3 new servers and were lucky enough to have another 2 servers donated.

As of January 2013, the OTW owns 11 servers and one switch to communicate between them, and pays for space on 3 virtual servers (some of these will be decommissioned in the coming months).

Our physical servers are at colocation facilities. Our current hosting costs for our physical machines amount to $1,640 a month, and we pay another $370 a month for virtual hosting.

Planning for expansion

Systems have spent a lot of time thinking about how to manage the current demand and plan for continuning expansion going forward. We have put in processes which will allow us to add servers to the organization with efficiency so that additional growth can be done with far less systems administration work that has been required in the past.

One of the issues we have had in the past is that the systems have been maintained individually. When the number of servers is small, this can be maintained but it becomes unwieldy as more servers are obtained. We have been trying to balance looking after the servers with committing to the work needed to automate the installation and configuration of the systems needed to provide the Archive and Fanlore and the other org sites.

Over November and December 2012 the group spent around around 25 hours a week automating the installation (with fai, the fully automated install system) and configuration (with cfengine3, ). Once this work is complete we should be able to provision new servers both physical and virtual quickly and consistently.

Machine specifications

As of January 2013, the machine specifications and jobs were as follows:

Definitions:

Virtual machine: a server that looks like an actual computer but is actually software built on top of a larger, higher performance server. Virtual machines are ideal for web servers and other basic workhorse systems.

Storage device: A system that is mostly disk space and networking. Imagine a gigantic external hard disk times a billion.

Service machine: A system that runs mostly behind the scenes programs that Joe User never sees, but OTW staff may need.

KVM: Keyboard/Video/Mouse: Servers generally do not come with these, but are just big boxes full of disks, memory, CPU, and lots and lots of fans. To talk to a server directly, while in front of it, you generally need a KVM.

DNS: Dynamic Name Service. What tells other computers (like yours) where to find sites like archiveofourown.org.

Colocation: Remote hosting site where servers are kept. Hosting costs include power, cooling, and someone to physically work with the machine when needed.

The new servers have a total of 48 cores of compute, 768GB of RAM, and 800GB of very fast storage. We are currently providing the Archive on 28 cores of compute and 192GB of RAM.

Archive of Our Own - server setup

As of January 2013, buying new servers had allowed us to significantly restructure the systems architecture for the Archive of Our Own. The technically minded can see the basics of our setup below (this is a simplified version of the current configuration with systems such as the mail server and the secondary mysql servers removed:

January 2013 - our server setup

Over the course of 2012, the demands on all our sites grew. In particular, the number of users accessing the Archive of Our Own each month dramatically expanded from 963,818 in January 2012 to 2,970,103 by December 2012. This demanded a significant expansion of our hardware: we bought 3 new servers and were lucky enough to have another 2 servers donated.

As of January 2013, the OTW owns 11 servers and one switch to communicate between them, and pays for space on 3 virtual servers (some of these will be decommissioned in the coming months).

Our physical servers are at colocation facilities. Our current hosting costs for our physical machines amount to $1,640 a month, and we pay another $370 a month for virtual hosting.

Planning for expansion

Systems have spent a lot of time thinking about how to manage the current demand and plan for continuning expansion going forward. We have put in processes which will allow us to add servers to the organization with efficiency so that additional growth can be done with far less systems administration work that has been required in the past.

One of the issues we have had in the past is that the systems have been maintained individually. When the number of servers is small, this can be maintained but it becomes unwieldy as more servers are obtained. We have been trying to balance looking after the servers with committing to the work needed to automate the installation and configuration of the systems needed to provide the Archive and Fanlore and the other org sites.

Over November and December 2012 the group spent around around 25 hours a week automating the installation (with fai, the fully automated install system) and configuration (with cfengine3, ). Once this work is complete we should be able to provision new servers both physical and virtual quickly and consistently.

Machine specifications

As of January 2013, the machine specifications and jobs were as follows:

Definitions:

Virtual machine: a server that looks like an actual computer but is actually software built on top of a larger, higher performance server. Virtual machines are ideal for web servers and other basic workhorse systems.

Storage device: A system that is mostly disk space and networking. Imagine a gigantic external hard disk times a billion.

Service machine: A system that runs mostly behind the scenes programs that Joe User never sees, but OTW staff may need.

KVM: Keyboard/Video/Mouse: Servers generally do not come with these, but are just big boxes full of disks, memory, CPU, and lots and lots of fans. To talk to a server directly, while in front of it, you generally need a KVM.

DNS: Dynamic Name Service. What tells other computers (like yours) where to find sites like archiveofourown.org.

Colocation: Remote hosting site where servers are kept. Hosting costs include power, cooling, and someone to physically work with the machine when needed.

The new servers have a total of 48 cores of compute, 768GB of RAM, and 800GB of very fast storage. We are currently providing the Archive on 28 cores of compute and 192GB of RAM.

Archive of Our Own - server setup

As of January 2013, buying new servers had allowed us to significantly restructure the systems architecture for the Archive of Our Own. The technically minded can see the basics of our setup below (this is a simplified version of the current configuration with systems such as the mail server and the secondary mysql servers removed:

January 2012 - our server setup

At the beginning of 2012, the OTW owned 6 servers and paid for space on 6 virtual machines for the rest of our services. The Archive of Our Own was completely hosted on servers we owned (5 of the 6). This is important because owning the servers makes it easier for us to protect fanworks. We also had one switch for the Archive servers (this communicates between the different machines).

All our physical servers were at a colocation host - we pay them for space, electricity and bandwidth, and physical maintenance when required. The hosting costs at the start of the year were around $800 per month. The charges for the virtual servers were $420 per month.

Machine specifications

As of January 2012, the machine specifications and jobs were as follows:

Definitions:

Virtual machine: a server that looks like an actual computer but is actually software built on top of a larger, higher performance server. Virtual machines are ideal for web servers and other basic workhorse systems.

Storage device: A system that is mostly disk space and networking. Imagine a gigantic external hard disk times a billion.

Service machine: A system that runs mostly behind the scenes programs that Joe User never sees, but OTW staff may need.

KVM: Keyboard/Video/Mouse: Servers generally do not come with these, but are just big boxes full of disks, memory, CPU, and lots and lots of fans. To talk to a server directly, while in front of it, you generally need a KVM.

DNS: Dynamic Name Service. What tells other computers (like yours) where to find sites like archiveofourown.org.

Colocation: Remote hosting site where servers are kept. Hosting costs include power, cooling, and someone to physically work with the machine when needed.

Archive of Our Own - server setup

As you can see from the above, the Archive of Our Own uses the most servers and therefore has a more complicated server setup. For the curious (and technically minded) here’s how they were organised at the start of 2012:

January 2012 - our server setup

At the beginning of 2012, the OTW owned 6 servers and paid for space on 6 virtual machines for the rest of our services. The Archive of Our Own was completely hosted on servers we owned (5 of the 6). This is important because owning the servers makes it easier for us to protect fanworks. We also had one switch for the Archive servers (this communicates between the different machines).

All our physical servers were at a colocation host - we pay them for space, electricity and bandwidth, and physical maintenance when required. The hosting costs at the start of the year were around $800 per month. The charges for the virtual servers were $420 per month.

Machine specifications

As of January 2012, the machine specifications and jobs were as follows:

Definitions:

Virtual machine: a server that looks like an actual computer but is actually software built on top of a larger, higher performance server. Virtual machines are ideal for web servers and other basic workhorse systems.

Storage device: A system that is mostly disk space and networking. Imagine a gigantic external hard disk times a billion.

Service machine: A system that runs mostly behind the scenes programs that Joe User never sees, but OTW staff may need.

KVM: Keyboard/Video/Mouse: Servers generally do not come with these, but are just big boxes full of disks, memory, CPU, and lots and lots of fans. To talk to a server directly, while in front of it, you generally need a KVM.

DNS: Dynamic Name Service. What tells other computers (like yours) where to find sites like archiveofourown.org.

Colocation: Remote hosting site where servers are kept. Hosting costs include power, cooling, and someone to physically work with the machine when needed.

Archive of Our Own - server setup

As you can see from the above, the Archive of Our Own uses the most servers and therefore has a more complicated server setup. For the curious (and technically minded) here’s how they were organised at the start of 2012:

Happy Ada Lovelace Day from everyone at the Archive of Our Own!

The majority of AO3 volunteers are women, so we're thrilled to have the opportunity to celebrate women in technology! Our testers, coders and systems staff are all inspirational, and in the five years they have been working on the Archive they have all become role models for one another.

This Ada Lovelace Day we'd like to give a shoutout to one inspirational woman in particular: AD&T Co-chair Elz. Elz has been coding on the project since the very beginning, and we're in awe of her amazing patience, her great teaching and mentoring skills, and her ability to produce massive amounts of code: she is the number one contributor of code to the AO3 and has written over 130,000 lines of code!

Elz is particularly inspirational because she didn't originally come from a STEM (science/technology/engineering/math) background: she majored in literature at university and is a self-taught coder. The experience she gained working on the AO3 enabled her to move into coding professionally, and she now works in the tech industry: a great example of how sharing skills in the open source community can enable people from non-STEM backgrounds to move into technology.

Elz has done amazing work on the AO3 over the years, and she's been incredibly generous with her time and her skills over the years. We're happy to celebrate her on Ada Lovelace Day!

We'd also like to celebrate all our volunteers, especially those who joined us more recently: we've had fifteen coders make their first commits to the project in the last year. We're proud of the achievements of all our volunteers - of all gender identities - and we're grateful to them for their hard work.

Happy Ada Lovelace Day!

We've been receiving a small number of reports of people unable to access the Archive of Our Own - if you've been affected by this issue, this post will give you a bit more information about what's going on. We'd also like to appeal for your help as we work to fix it!

What's wrong

We recently upgraded our firewall to improve the security of our servers. Unfortunately, it seems that we haven't got the configuration absolutely right and it's causing connection problems for some users. This problem is only affecting a small number of users, and it's not completely consistent. However, if you've received an Error 404, a warning saying 'Secure Connection Failed', or you've been redirected to a url with 8080 in it, then this is what's causing it.

While we worked on the issue, we temporarily disabled https on the site, as that was causing some additional problems. This means that if you have a browser extension such as HTTPS Everywhere enabled, or you use a browser which enforces https by default, then the site will not load - apologies for this. If the site has been consistently timing out for you, it's worth checking if this applies in your case - if the url defaults to then you have been affected by this issue.

How you can workaround

If you're being affected by the https issue, you can work around by adding an exception to HTTPS Everywhere, or using a different browser.

If you're getting errors at random, then clearing your browser cache and refreshing should help. You may also find it helps to use another browser.

How you can help us

We're working to get to the bottom of this problem, and we know we've already reduced the number of errors which are occurring. However, it would be enormously helpful for our Systems team to have a little more information. If you encounter an error, please submit a Support request giving the following information:

Your IP address You can find this out by going to http://www.whatsmyip.org/.
The url of the page where you got the error:
The exact error you got: You may find it easiest to copy and paste the error. If you didn't get an error but the page just never loads, tell us that.
What time (UTC) you got the error: Please check the current time in UTC when you get the error - this will make it easier for us to keep track, since we're dealing with users in lots of different timezones.
Is the error intermittent or constant?
What browser are you using? It would be extra helpful if you can tell us your user agent string, which you can find out by going to http://whatsmyuseragent.com/".

If you know how to view the source of a page in the browser, it would also be very helpful if you could could copy and paste the source code of the page that throws up the problem.

If you're comfortable working on the command line, then it would also be helpful if you could provide us with some additional information (if you're already wondering what we're talking about, don't worry, you can ignore this bit). Open up a command line window and type nslookup ao3.org. Copy whatever pops up in your console and paste it into your Support message.

If you can't access the Archive at all (and thus can't submit a Support request there) you can send us this information via our backup Support form.

A note on https

We know that many people prefer to use https connection to provide additional security on the web, and we will be reenabling this option as soon as we can. Because the AO3 doesn't handle data such as credit card information or similar, browsing without https doesn't expose our users to any significant security risks. However, it is always a good policy to use a unique password (i.e. don't use the same username and password combo for the AO3 and your email account) in order to ensure that if for any reason someone else obtains your AO3 credentials, they can't use them to access other data). Apologies for the inconvenience to users while this option is disabled.

ETA for a fix

We're hoping to resolve these lingering problems asap; however, our Systems team have limited time, so we may not be able to track down the root of the problem as fast as we'd like. We'll keep you updated, and in the meantime apologise for the inconvenience.

Happy SysAdmin Day! Here at the OTW, we'd like to take the opportunity to say thank you to our fantastic Systems team!

Systems work tirelessly behind the scenes to make everything work smoothly for the whole organisation.

What you see

What Systems do

Maintain the servers for the Archive of Our Own, Fanlore, Transformativeworks.org, Transformative Works and Cultures, Open Doors, and our internal wiki; install server software; arrange the installation of new hardware; find solutions when Fanlore is hit by a wave of spam; optimise performance on the AO3; wake up in the middle of the night to fix things when our servers melt; maintain and administer web development environments for our trainee coders; research and consult on hardware purchases; answer endless technical questions so Communications can post and tell people what's happening; pull stats to help us understand more details about our projects; set up new software tools; and much, much more.

Because of Systems, fandom can own the servers!

Our Systems committee are super-duper awesome and make it possible for all our projects to exist! <3 THANK YOU for your awesome work and all that you do!

Go go Systems monkeys!

The OTW Board announces with regret the resignation of Sidra, the co-chair and technical lead of our Systems committee, who has been one of our senior technical staff members from our early days. She has supported virtually all our projects with her vast technical skills and immense generosity with her time, and has been one of the foremost contributors to the Archive of Our Own.

Sidra is staying on as a consultant with Systems, so she is not vanishing completely, but we're hugely grateful to her for all her work and want to thank her publicly for her years of incredible service to the org, and we hope all our members and the users of our varied projects will join us in those thanks!

Mirrored from an original post on the OTW blog. Find related news by viewing our tag cloud.

Happy Ada Lovelace Day from everyone at the Organization for Transformative Works!

Celebrating women in technology is a subject close to our hearts: when the OTW came into existence in 2007, one of our major motivations was the desire to give fans control of the tools and infrastructure which support fannish creativity. The predominately female fannish communities from which the OTW emerged have a long history of mastering new skills and sharing expertise for fannish pursuits — the vidders of the 1970s were pioneering mashup techniques decades before they became trendy! — and we want to extend that skill-sharing to the creation of a fan-owned home that welcomes all fans.

The vast majority of OTW volunteers identify as female, and the amazing things our teams have achieved demonstrate that they all deserve to be considered tech heroines! Below, we highlight the work of our tech-focused teams and the individual voices of some of our staff and volunteers.

Archive of Our Own

The AO3 is the major tech project for the OTW, and is supported by several committees and volunteer groups: Accessibility, Design, & Technology; Systems; Support; Tag Wranglers; Coders; and Testers. We're one of the largest female-majority open source projects in existence, and we're proud that in less than four years we've developed from nothing more than a cool idea to become a thriving site with more than 23,000 users.

Last Ada Lovelace Day we polled AO3 volunteers to find out a bit more about them, and we thought we'd repeat the experiment this year. The charts below give a summary of their answers:

We're still very definitely a female-dominated project; however, we're interested to note that since last year the number of volunteers who identify as male has increased by 10%. We think this reflects the fact that we are focused on making a welcoming and supportive environment for people to gain new skills. As Skud pointed out in hir 2009 Oscon keynote, making a project welcoming for newbies is particularly beneficial to women — who are often excluded from traditional tech contexts — but that doesn't mean it becomes less welcoming to people who aren't women!

Not all the contributors to the project are coders or sysadmins; the AO3 also relies on the work of testers, tag wranglers, support staff, designers, and docs writers. We value their contributions just as much: a tech project is about more than lines of code, and without them the AO3 wouldn't exist.

A key part of our goal is giving fans (whatever their gender identity) the skills to build the tools they want to use. We were super-proud to see some of the fruits of this mission during the recent Delicious debacle, when fannish talk quickly turned to "We should build our own bookmarking service — if the AO3 could do it, so can we!" Our volunteers have achieved so much — they're all tech heroines (and heroes)!

The AO3 team would like to give special thanks to one particular tech heroine — Sidra, Systems co-chair and primary guardian of the servers for the AO3. The Accessibility, Design, & Technology Committee have posted a separate post celebrating Sidra's awesome work.

Fanlore

Another major technical undertaking for the OTW is Fanlore, our fannish history wiki. Since Fanlore is built on existing MediaWiki software rather than a custom-built application like the AO3, the tech aspects of this project are not as immediately obvious, but they are just as important. Our Wiki staff have learnt to maintain and use the MediaWiki software, creating custom templates, investigating new software modules, and getting to grips with wiki maintenance. They are awesomely assisted by our Systems team, who installed the software on our servers and keep everything running smoothly (we love you, Systems ♥).

Fanlore is celebrating Ada Lovelace Day with a new challenge on Women Characters, Science Edition! Why not create a Fanlore article about your favorite female character who is a scientist, engineer, or mathematician? Tell us about your fannish experiences with these characters — the women themselves, the relationships they’re in (het, lesbian, canonical, fannish, etc.), the fanworks they star in — whatever you can think of! You can stub out a new page, or add a sprinkle of information on an existing page.

Systems

If you've read this far, you've probably realized that Systems is involved in every OTW project. They tend the AO3 servers; install software for Fanlore, Transformative Works and Cultures, Open Doors, and the main OTW website, plus the software that helps us process donations and manage volunteers; and set up the mailing lists that help all the committees and volunteer groups do their daily work. The heroines and heroes of the Systems committee work largely behind the scenes to keep our technical infrastructure running smoothly, and the entire OTW benefits enormously from their dedication and expertise.

Webmasters

The Webmasters are another committee whose work is spread among a wide variety of projects. They maintain the OTW's main website, the Open Doors site, and the Elections site, manage our donation processing software, serve as layout coders for Transformative Works and Cultures, design styles for the OTW's social media accounts, and manage media hosting for various internal projects. To date, the Webmasters have all been women, and have been largely self- or peer-taught in the technical skills they use.

Some thoughts from our volunteers

In a post that celebrates women doing it for themselves, it seems appropriate to close with some thoughts from our volunteers, as they reflect both on their own work and on that of other women they admire. We'll be adding links to individuals' blog posts at the end of this post throughout the day.

It's exciting to work in teams that are overwhelmingly female. I really like the testing parties, as it's a little confusing and intimidating to try to work from written descriptions. I joined to support an organization I trust and approve of, and to get some practical tech experience. I just started volunteering a few weeks ago, so not much to say yet!

Sometimes I have conversations about servers, code, etc and I realise that former!me wouldn't have understand ANY of it. I've only learnt enough to contribute a tiny amount of code, but I am able to be a fully functioning member of AD&T because I have absorbed enough to be able to take part in these conversations as a useful laywoman.

I like finding interesting bugs and feel good whenever I find one before it hits Beta.

I like that the archive tries to accommodate a variety of people and systems instead of saying: get browser x with y settings or we don't care about your problems.

I love wrangling big fandoms with lots of problems and characters-shared-between-fandoms, it's a big undertaking but it's nice to see everything all neat once you're done!

Since I come from a background of relatively no coding, it has been really exciting to submit my bug fixes and see my changes on the archive! The whole experience has been really rewarding!

Since beginning my work with the Archive, I have improved my computing skills dramatically. I have learned a great deal about linux and switched to a more complex, text-based distro. I have gained an exceptional amount of skill and confidence with unix commands and bash. I now have an understanding of how the Archive is put together via Ruby on Rails, and that understanding deepens and develops with every issue I work on. This has been an amazing experience and I am excited to keep learning and growing as a coder!

I've never been part of a mainly women-identified group before, and it's really been rewarding for me in so many different ways. I'm so proud to be part of the OTW!

It combines two of my dearest hobbies: Coding and fandom. Both Open Source people and fandom people build great, communicative communities with lots of collaboration, and if you put those two together you get fun squared. :D It's really great to share more than the passion for coding with my fellow coders, so when I'm in a phase where I code less in favour of writing or squeeing over a new shiny fandom, it's never really off-topic, thus making it easier to keep in touch with coding stuff.

[Something I'm proud of accomplishing.] Dragging a committee up from its bootstraps at the project's launch, in such a way that it perfectly well survived (and prospered after) my own burnout-related crash and burn.

I really love it. I quit grad school in a blaze of disillusionment and have been unemployed and completely at sea in my life since, and it's been really heartening to have something I can contribute to in small ways, especially something that's part of fandom, which has been such a wonderful aspect of my life for so many years.

It is one of the more nurturing and family-building projects/organizations I've seen.

It's a delight to work on a project where people not only don't jump to assumptions about you, but where people are supportive even if you make the smallest contributions.

ruby metaprogramming! redis! There is just nothing quite so fantastically satisfying as working with a smart and dedicated and passionate team on a project that we all actually use ourselves and value deeply as a result.

I've really enjoyed being AD&T training lead, running sessions for new people to learn how to code from scratch, and mentoring them as they advance. It's so rewarding to see people gaining new skills, and particularly when you know they've previously been excluded from opportunities because of their gender or disability, e.g. by lack of part-time courses that can fit around childcare or flare-ups.

I'm *so excited* to be part of the team that's creating the Archive that I love so much. I think fandom is amazing to have worked so hard together to create the Archive.

Mirrored from an original post on the OTW blog, where we'll be collecting links to Ada Lovelace Day blog posts from OTW members throughout the day.