@dachary: So it's on. So regarding where the transcript and how it will be used, the intention is to use it in what we call affinity mapping, which is essentially we slice the interview to remove the narrative and only keep assertions that is phrases that you say, that state a fact or an actual experience. But we don't keep the story part of it, because what we are looking for is evidence from the field, from actual experience about problems or just things that are done in a certain way and it is these evidence that gives us legitimacy to try to implement something, to address what has been said. As opposed to just imagining that there is a problem or that people do things in a certain way where in reality they don't. So that's the goal. So I will start by asking questions about you as a participant in the organisation so can you maybe tell me a little bit about your background as a sysadmin.
@3wc: Yeah, absolutely! I've been working on systems administration mostly on Linux and a little bit of Windows for 10-15 years. Depends exactly where you start. Within Autonomic cooperative, I'm one of the founding members and I've been somewhat involved in the infrastructure over the years. We initiated a project called Coop Cloud, for which we also host a Gitea instance. I set-up the Co-op Cloud one, and I help a bit with the maintenance of the Autonomic Co-operative one.
In terms of general background, I knew Ansible before encountering Gitea. We currently deploy Gitea using Docker, which I learned through working on Coopcloud. I wasn't familiar with it before we had, it was my colleague that setup a Docker based Gitea instance for us -- now I understand the setup well enough to maintain it but at that time Docker was entirely new to me and didn't have any experience with that. I've run Trac and (Phabricator](https://en.wikipedia.org/wiki/Phabricator) at a previous job, and maybe a couplle of other small code forges, and I've used SVN, CVS, Mercurial as a developer.
@dachary: And your first installation of Gitea was for Autonomic via Docker, you didn't do it before?
@3wc: Exactly! And I've also setup GitLab CE for the previous employer but we didn't end up using it extensively. That is sort of general overview, is there anything else in particular sysadmin-wise that you'd be curious about?
@dachary: yeah, I'm always curious about how you choose to work on something. So you are in a cooperative where you have more leeway than a usual company but how does that happen?
@3wc: Maybe it would be useful to tell you about the process to arrive on Gitea.
@dachary: Sure!
@3wc: When we started Autonomic, we had no capital to speak of and our client income was growing slowly. The amount of time and money that we had for own infrastructure was minimal. So we started out on GitLab free account and I think we signed up long enough ago that they gave us quite a generous free plan and this was before the set of serious ethical issues with GitLab came out. And I think before git.coop was widespread through the Cotech network so we chose GitLab without knowing either of those things
@dachary: I see
@3wc: while using those hosted services, we tried to have a plan to migrate into self-hosted infrastructure. Partly because of values and partly because we are trying to host infrastructure for others and we need to build those skills. Our impression of Gitlab was that was very heavy to self-host and complicated. And also none of us particularly liked the interface, we found it slow and confusing and we are still currently using GitLab for collaboration with one of our clients that was already using it, and we are still getting confused by the interface.
So that was one of the huge advantage -- there were really two things that led us to Gitea, or led my colleague towards Gitea a couple of years ago when we first set it up. It is incredibly simple to run, and it is really fast.
The interface is a lot more straightforward, it is more familiar to GitHub or the other kind of ticket tracking systems we've used. The navigation has fewer layers and we were all impressed with how quick the interface was, even though it is not doing a lot of AJAX requests, many of the navigation operations involved full page loads, it's still much faster than GitLab in our experience. And I've also noticed the same with the Git operations commonly when I'm working with my colleague, it will take like 5-10 seconds for them to be able to push something up to GitLab whereas Gitea is pretty instantaneous. And then, Gitea has sort of been our test bed because it was the first application that my colleague setup using our -- first attempt at the tech stack for Coopcloud which was using -- have you ever heard of Dokku?
@dachary: yes
@3wc: We used it as sort of open source Heroku. So we had a version of Gitea that was initially hosted through that. My colleague was experimenting with using that as a framework for hosting a variety of apps. We switched, I think, at the beginning of last year to Docker Swarm -- maybe slightly longer than that, maybe mid-2020, we switched to Docker Swarm and we built Coopcloud on that. I think Gitea was the first application we packaged using that format, and is the longest running sort of test bed. And that's still how both https://git.autonomic.zone and https://git.coopcloud.tech are running using that Docker Swarm configuration.
@dachary: Yeah, that is very helpful to know how you came to what is currently in place. You mentioned that initially you had an installation, you were looking for forges for your clients and you started using GitLab but it was too heavy. Now you are using Gitea, can you give a rough number of clients for which you have -- you manage Gitea instances?
@3wc: That's an interesting thing about our current setup: we only have two Gitea instances and one of them is for Coopcloud and as soon as we were successful in starting Coopcloud federation then that will, I guess, no longer be Autonomic hosted. All of our client projects are on our main Gitea instance. We don't yet have a client who needed their own private forge instance. We've used private organisations and private repositories within our existing instances for hosting clients.
@dachary: And so you say it's interesting because you seem to say that initially you thought you would create instance for your clients but it turned out not to be the case? Or did I misunderstand?
@3wc: I don't know if we knew what the major need would be for git hosting. No client has approached us with the combination of needs that added up to they have to have a private Gitea instance. Also, you wanted to ask about our experience maintaining Gitea, and I think there was a little bit more fragility than we were expecting, that I think scared us off running multiple instances but I don't know if I certain that we definitely had intentions to run many instances right from the get go. I think it has all been a bit experimental.
@dachary: I get the impression because this you've done a wonderful job having generic Ansible stack that can be self-hosted, etc. So it suggests that you were into self-host for the benefit of clients that's probably from there(!verify: 11:10)
@3wc: It's a really interesting observation. I think I'm stumbling over my words a bit because I just haven't really thought about it before but -- actually there have been two software development projects where it was a software co-op that approached us and was considering getting a Gitea instance.
I think it was a combination of us having not found it that easy to host for a couple of reasons that I will get into and the existence git.coop and other kind of decent hosts.
I think the main thing that has been on our minds as what we've wanted to do with Gitea hosting is offering individually. So it -- either as a benefit to members, or as a kind of service to compete with people hosting their personal stuff on GitHub or GitLab. So, like a software as a service model.
@dachary: So you say that one of the reasons maybe why you don't have self-hosted Gitea instances managed is because you found that it's fragile or you had experiences that scared you off?
@3wc: Yeah, exactly! And the first thing I'll tell you that the thing has improved over time. The main issue, that was the longest-running and caused the most time taken up is that until few months ago, Gitea had an issue where if it started and a single sign-on(SSO) provider wasn't available then it would disable that single sign-on provider so you'd have to login as the Gitea administrator and re-enable it but that's now been fixed. The remaining issue that that's plaguing us is because we haven't spent the time to dig into is we are not setting session cookies correctly. So every time there's been a restart, everyone gets logged out.
And then there's a conceptual issue which raises the bar for hosting and I'm not even sure what this solution would be, I don't know if this is something that you've considered for your effort or if it's why you are learning towards VPS' over containers for Gitea hosting is that our current model is to pack apps quite densely onto single VPS, which works for large majority of apps that as soon as an app needs network service that doesn't do server name indication, for instance SSH as faras I know doesn't do server name indication, so you can't have a reverse proxy listening on the SSH port and directing requests to the appropriate container without having so kind of additional layer on the SSH protocol. I think you do that with things like stunnel but then requires configuration on the client-side.
So there's a limitation -- what we currently do is we have a different port for connecting to the server to manage it than we do for Gitea's own SSH server. The problem is that we would then need to do some manual reconfiguration for every additional Gitea server that we added to that to be able to use port 2223, 2224, whatever for each of the additional servers. And that means that if a client approaches us and says, "we'd like our own Gitea server" then we'd either need to say, "Okay, but you can't do SSH" or that we need to invest the time and set-up and maintenance annoyance in adding another port to our reverse proxy configuration which is frustratingly difficult. And I remember some other flakiness in towards the beginning of using Gitea around upgrades, I think the switch from the normal to the rootless deployment was a bit of a fiasco which happened sometime last year but yes, that's sort of what I meant in terms of fragility in the kind of friction.
@dachary: ah, I see and more recently there has been issues with the 1.16 with the 2FA transition to webauth which maybe you don't use so you don't notice but people noticed. So yea, I clearly see what you mean. Okay, this is very interesting. I will move to our series of more precise questions regarding various topics and starting with contract and pricing. So I understand that your client do not -- you do not bill your client for a specific slice of the Gitea hosting but it's part of the service you provide them? Did you put price tag on that, more or less, even internally?
@3wc: The short answer is no, the slightly longer answer is that we have one organisation that is making a monthly donation in order to use git.coopcloud.tech, I think that maybe have three repositories, two users or something like that and they are giving us, I believe, £10 a month for the hosting.
@dachary: Okay, well that's an interesting -- that's a number (!verify: 17:18).
@3wc: All of the others, as you say, it's just rolled into the overheads of offering sort of software.
@dachary: So you mentioned that there are -- how many clients your use your Gitea instance? A rough number (!verify: 17:38)
@3wc: There's four clients accessing it directly and we have code for a further three on our Gitea instance.
@dachary: Okay. So interesting that you have clients who are aware and using it and your other clients where you use it for their benefit but they don't use it, correct?
@3wc: Exactly, yep.
@dachary: Okay. So for the clients who, so this is moving to the technical and support part of the questions, so the clients that actually use Gitea: how do they report issues to you, how does that happen?
@3wc: Thinking. It's been a combination of us entering issues based on them emailing us a list of requests or a file with a lists of requests and on one hand and on the other hand, clients entering issues directly and us triaging. It varies with different clients.
@dachary: Okay, and you say entering issues directly: are they using Gitea to enter issues or other...
@3wc: Yeah
@dachary: Okay. And they are using your -- their own Gitea repository to file issues or you have a dedicated one for client issues?
@3wc: The client -- there's one client on each of our Gitea instances that's directly entering issues and they both have, in those respective cases, one repository they use. They file issues directly in those repositories. So we don't have a separate issues-only repository, it's in with the code.
@dachary: Okay yeah, fine. And as you say it's only one person who enter the issues themselves. So you don't have a generalized system for all the needs of all the customers. It's a case by case basis, correct?
@3wc: Exactly! I would say that we have a little bit more defined practice around issue entry and triage systems for the client where we still use GitLab, because those clients were using it already.
@dachary: Okay, I see. So you mention it that I will repeat it, just in case I didn't listen correctly, when they have feature requests or an improvement request, they also file an issue. It's the same for problems and feature requests
@3wc: Yes, exactly. And then we label in Gitea to differentiate between those.
@dachary: Okay, fine. Correct. Moving on to the really deep, technical things: could you describe step-by-step, any level of detail that you prefer, how the installation process of a Gitea instance goes? How would you guide someone who wants to do it using your stack?
@3wc: Yeah. So there's a bit of a decision tree that occurs into it, which the main question is whether they need a separate VPS and if they do, we would setup a new VPS using Ansible playbooks which would configure the file, and install Docker, setup user that kind of thing(a SSH user account, that is Unix users) and then once that's done or if we decide that there is a suitable server running already, for instance for a couple of clients we already run a VPS for them with some other Co-op Cloud Docker swarm apps, so in those situations it might make sense to setup Gitea on that.
So the server is set-up. For the app, the screencast I sent in Matrix runs through the process. To very briefly summarize that: now, you'd run a series of commands using abra, which is the Co-op cloud command-line interface. Within Autonomic, the standard is to run that on one's own workstation and to interact with a shared repository or environment variable files representing different application. So at Autonomic we have a single Git repository on our Gitea, representing all of our clients, our cloud app configurations. If you run commands to create a new Gitea instance, it will create a new file in that directory which we check into Git, push out to the central repository. And then we'd run some further commands which would connect via SSH to the designated virtual server and deploy the Gitea instance by downloading Gitea, MariaDB, whatever the Docker image is necessary, starting them, setting configuration based on environment variables that we defined along the way and then going into the Gitea user interface to configure a single sign-on and...
@dachary: may I briefly interrupt to (!verify: 24:03) what I understand
@3wc: Yes, please!
@dachary: So from the workstation, you run abra and then abra will -- you designate a VPS where Gitea has been installed and abra carries environment variables and will download Gitea so it's all abra doing until you arrive at the web interface of Gitea, is it correct or is there some minute steps automated via
@3wc: Exactly!
@dachary: And then
@3wc: By the time you arrive at the user interface, it will have provisioned a user using a secret that is displayed, it's generated and displayed during the abra setup process. So the default is a secure setup. It's not like Wordpress where when you finish an install anyone can go to the URL and complete the setup.
@dachary: Okay. So then you go to the web interface and...
@3wc: Yeah, I mean this is one thing that I did want to touch on. I will say Gitea seems the best, the easiest of any of the tools that we used to configure single sign-on providers. The fact that you can do it in the user interface and that it is very clear how you can add different authentication providers. We tend to use Keycloak for single sign-on, so there would be a couple of steps on the Keycloak side to generate an open ID client for Gitea, to get the client ID and the secret for the Gitea side of the configuration and I'm trying to think if there was anything else -- oh, and the other thing that we commonly do is setup a Drone instance to provide CI/CD for Gitea and that we would also through abra and the similar process of committing the generated application environment variable files to Git.
@dachary: Okay. And when was the last time you deployed a combination of Gitea and Drone, for instace?
@3wc: I think it was when we setup https://git.coopcloud.tech/, which was July last year, something like that.
@dachary: Okay. And side question: did you consider Woodpecker?
@3wc: I don't know what that is
@dachary: It's a fork of Drone. Okay so you did not.
@3wc: Before the license change, is that correct?
@dachary: Honestly, I don't know the history. I just know that Drone is not very actively maintained and Woodpecker is very lively at the moment
@3wc: Interesting!
@dachary: It is mostly compatible but not quite. Anyhow, that was a side question. Okay. And
@3wc: I think I heard about it briefly through Cyberia, because I think they were considering it. They are currently using concourse and they heard we were using Drone and they suggested a fork, we haven't looked into it.
@dachary: Okay. And I suppose in the process of this installation there are also backups?
@3wc: Yes, we tend to enable server-level backups on the VPS as one strategy and then -- I don't think it is supported for Gitea yet but we are rolling out application-level backups for Coopcloud apps, which will give you like a archive of the files directory and an archive of the SQL dump in a convinient format rather than have to restoring an entire operating system image.
@dachary: So when you say a VPS-level backup, it's like you have the copy of the QCOW image or something like that?
@3wc: Yes, exactly! Where we are hosting it on our own infrastructure. Where we are using a commercial VPS provider, then we use their snapshot features.
@dachary: Okay, correct. (!verify: 29:58). What about monitoring? It's also done by abra's side effect of the installation?
@3wc: It depends, there's a level of service monitoring that is provided by Docker Swarm in terms of automatically restarting failed processes. We are currently rolling out a stack based on Grafana and Prometheus, haven't been involved since setting it up so I'm not so confident on the details, for monitoring Coopcloud apps. On top of that, we also use UptimeRobot for ping and uptime monitoring.
@dachary: And you have any kind of intrusion detection setup? Something that would check and give (!verify: 29:53) files, that kind of things?
@3wc: We've used lynis and I've used the rkhunter, can't remember if we've used that app in Co-op on our servers but we don't have an automated scanning tool. We are looking into log file monitoring with Loki which we are hoping might give us some level of that.
@dachary: Okay. And last question on the setup: can you tell me a little bit about how the mail is setup for the Gitea instance? How does the Gitea instance connect to outgoing mail?
@3wc: Yeah, I'm just trying to login and check...
@dachary: It's fine if you don't remember
@3wc: It's going to be one of two options: either we are using a mailbox provided by domain host Gandi, or or we are using our self-hosted mailing stack and I will tell you in one second...
@dachary: And this is taken care of by the recipe, that abra uses to deploy Gitea, is part of the automated install
@3wc: Yes, there's a step during the abra setup where you are editing a configuration file that is specific to your instance of Gitea, and at that stage you'd define what the SMTP server and any credentials you need to access it were.
Our instance is using a Gandi mailbox, we do also self-host a SMTP server on the same server. I can't remember why we didn't use that for Gitea. We would probably choose one or the other depending on what (!verify: 32:05)
@dachary: Okay, Great! That concludes all the installation process set of questions. I have a few questions about maintenance. So you mentioned having VPS' to run Gitea and you also have a Drone instance, what is the size of these VPS'? Are there two different VPS'?
@3wc: Main Autonomic Drone is running on the same server as our Gitea and that server is a quad-core with 16 GB RAM and...
@dachary: So it is a physical machine?
@3wc: No it's just a tremendously expensive VPS. And the Coopcloud Gitea is also running on the same VPS as the Co-op Cloud build server, that is a 4GB RAM three core VPS that is on our dedicated hardware.
@dachary: Excellent! Can you describe how a desaster recovery scenario would look like? For instance, say the VPS just dies, what happens?
@3wc: Specifically for Gitea?
@dachary: Yeah, yeah of course! I mean it's the same as every other because it's VPS based, so...
@3wc: Yeah, so you say the main server that Gitea is running on dies, on Autonomic Gitea, we'd provision a new VPS using our Ansible playbooks and...oh, actually no! I guess, because we -- That's a virtual server, we just use the features in our VPS host, Hetzner, to provision a new VPS from a snapshot backup before it died. The Coopcloud instance that's -- as I said, we are running a dedicated server to host the VPS server, as you said, we have a virtual image disk backup so we'd use our backup tool, I think it's restic on that, to restore that disk image to an earlier version.
@dachary: Okay, good. And you are -- you rehearse these kind of disaster recovery?
@3wc: Not deliberately though, it happened more than once, we deleted the wrong server by mistake so...
@dachary: Okay, I see. Yeah. But this is good because I'm always concerned about disaster recovery not happening often. To your clients, what kind of SLA do you tell them, for the clients that use your Gitea instance? What kind of SLA do you promise them?
@3wc: We have SLAs for some clients, we do GItea stuff with some clients but as far as I know, there is no overlap between those two groups. Just because the two clients that are most concerned about uptime don't have Gitea with us because, in one case they were already on BitBucket for several years and had no desire to move and on the other case, they don't host much code and I think they are currently hosting on Gitlab.
I think we might have a default SLA on our website, I'll try to find that now but I can also tell you what our actual actual uptime is
@dachary: It wouldn't be a separate SLA, it might be a part of other services' SLAs
@3wc: Right, yeah. I don't find it on our website. I'll just quickly tell you what our Gitea was because that might be useful
@dachary: Okay, sure!
@3wc: Doesn't seem like uptimerobot is monotiring this actual Gitea instance, can that really be that case? I get actual monitoring of other things on the same service. That's an interesting gap. Okay, I can't tell you the uptime, sorry.
@dachary: Okay, yeah. If you are using it multiple times a day, you are the monitoring service! In terms of personalization -- so you have these two instances, I saw that you added a logo, you changed a few bits -- could you describe what kind of personalization you did as for as you remember and how you did it?
@3wc: I -- yeah, I don't think we did much at all! There's some really urgent personalization things that we'd like to do, that login experience with single sign-on is really confusing and apparently the way to fix that is through a template change but non of us have found time for that. So every time you need to sign into our Gitea (which again is frequently, because it logs us out every time it restarts), you need to click a button at the bottom of the page which says "OpenID", there's also another button on the login page that says "OpenID". SO especially for clients that we've tried to get to use the single sign-on, it's been incredibly confusing. Right now, I don't think we necessarily changed the logo on ours -- I think the only customization is that we've changed the tagline to "Git with solidarity" instead of "Git with a cup of tea".
@dachary: Interesting! And, so you mentioned a few times that it's annoying that sessions expire when you reboot the server, how often do you reboot the Gitea instance?
@3wc: We at least weekly, or almost always atleast weekly because we install apt updates and it's often a Docker update which requires a restart of DOcker containers or a kernel update which restarting the whole server. There's also -- oh yeah, and then it's slightly more frequently than that because then, as well as that, there is also Gitea updates, which require -- which end up invalidating our sessions. I'm sure you've even seen the option to fix this in the documentation, just none of us had the time. Probably I could have fixed it in the time it took complaining about this on this call.
@dachary: When you mentioned that, I assumed that the accumulation of reasons to (!verify: 40:49) more than once a week, so you (!verify: 40:55) your two Gitea instances that you have Drone associated with it, are there other services also associated with Gitea, like Gitea uses the static page generator, developed by Codeberg or maybe some other third-party service? Or is it just...
@3wc: I only found out about the static page generator last week, I'd really love to use it. As far as I know, we are not using any services that integrate with Gitea. We do use some Gitea webhooks stuff to, yeah, Matrix notifications for activity in the Coopcloud reposts. I don't know if that counts, that's one integration that we have.
@dachary: So that would be a link to the chat in Matrix, from the top of your head do you remember another one, another webhook that...
@3wc: That's the only existing one currently -- oh no, I'm sorry there's a bot in the same channel, which I'm about to send you a link to, which you can use to create issues but, for the life of me, I can't remember the syntax for it. But it's a lovely idea, I love the concept. I can just never remember...
@dachary: Yeah, same here: I love to learn to use it and then I forget. That -- not specifically this bot but that kind of bot. That is something that is very attractive, somehow it does not stick. I don't know why. Now I have two last questions: one, which is a little bit bizarre so please indulge me!
@3wc: I'm intrigued!
@dachary: Imagine you have a magic wand and so you have to step back from your practical, very technical self and say you can use this magic wand just once and you can wish for anything as long as it is related to Gitea managed hosting so you wave your wand and you have your wish, what your wish -- what would that wish be?
@3wc: Just to make sure I understand, anything to do with managed Gitea hosting?
@dachary: Yeah, in this, yes
@3wc: This could be any, any like technical change to Gitea or any existence of the organization providing it or...
@dachary: We can be really wild as long as it is connected to this, to Gitea managed hosting, it can be...
@3wc: I promise I'm not just saying this because I know you are involved in it but - federation! Like that's the number one thing making it easier for someone on one Gitea instance to interact with content on another Gitea instance to lower the cost of moving off of GitHub or GitLab would be the killer for me. That, as a user, as an administrator and as a person trying to maybe make it somewhat financially sustainable for hosters like that is the number one.
@dachary: Oh, interesting! Okay, that's -- yeah, obviously I like the answer. I would have the same wish, I guess..
@3wc: And then, number 2 is like really only factor until number one is solved, but they way that we've worked around number one not existing is to turn on, open, GitHub and GitLab sign up on our Coopcloud Gitea. So that we could say that we are not on GitHub or GitLab but you can still use your account to sign in and it will be almost as easy as you were interacting with us on GitLab or GitHub.
@dachary: Yeah, it's one step
@3wc: Yes, it's one step less but it's still more than would be ideal. And we had to turn that off because we had an instance with someone logging in via GitLab and using our Drone instance to mine cryptocurrency. And I guess, I'm aware that there are some spam prevention features built into Gitea and it's a lot better than what it was but what I would love is some very accessible, i.e please not a CAPTCHA, default option for anti-spam. Because I know there is a CAPTCHA in the default distribution that, we could just turn that on but when we did previous usability research with people using screen readers, they told us in no circumstances should we ever use a CAPTCHA for anything. So I haven't felt comfortable switching that on so we've just been in this limbo now where you can almost sign in with those providers but you get to this frosting error screen in the end because open registration's disabled. So that would be a number two but obviously if you ditch federation or once federation happens then the need for that kind of anti-span would be less because we could just make our instance invite only again it would still be accessible to a wide community of people through federation.
@dachary: Yeah, sure! It has other spam issues but it's different. Okay, wow thank you! That was the first question. The second one is much easier, which is, if you had to think about one thing that you find works really well for how you managed Gitea hosting works, what would that be? If you had to talk to someone, "Oh we do managed Gitea hosting and this is something that we really like".
@3wc: I feel bad saying it but Drone! I think the top of the list would be Drone, it is really easy to set up, to configure both for the administrator and the people trying to do CI pipelines. It is incredibly fast, the output it gives you is tremendous. It's conceptually simple, there's no separate accounts for it. It just uses the same authentication as your Gitea. It's blown us all away, I think there are people who wouldn't be down for doing continuous integration or continuous deployment at all within Autonomic were it not for how good Drone is.
@dachary: Wow, interesting! I'm also very happy about Drone, really, and Woodpecker works the same ways. So it's just another name for the fork of the project. So, yeah. I completely agree, I'm also very happy about it. And, so...
@3wc: I think that's the worst part of GitLab's interface in particular. The worst part of GitHub's interface is the continuous deployment stuff as well. I don't know why they've made it such a hot mess.
@dachary: Yeah, same here, it's easier. So we are done with the interview. I thank you very much and before we wrap it, do you have any question? Last words?
@3wc: No, I thank you for doing this practice, in trying to build this offering. Like you said, it's not as common as it should be, I'm really interested in how you approached. I think it will hopefully improve any usability I'm doing in future. Yeah, I'm super excited to see what comes out of this project
@dachary: Thank you! I will cut the recording and...