diff --git a/3wc.md b/3wc.md new file mode 100644 index 0000000..516a281 --- /dev/null +++ b/3wc.md @@ -0,0 +1,649 @@ +@dachary: So it's on. So regarding where the transcript and how it will +be used, the intention is to use it in what we call affinity mapping, +which is essentially we slice the interview to remove the narrative and +only keep assertions that is phrases that you say, that state a fact or +an actual experience. But we don't keep the story part of it, because +what we are looking for is evidence from the field, from actual +experience about problems or just things that are done in a certain way +and it is these evidence that gives us legitimacy to try to implement +something, to address what has been said. As opposed to just imagining +that there is a problem or that people do things in a certain way where +in reality they don't. So that's the goal. So I will start by asking +questions about you as a participant in the organisation so can you +maybe tell me a little bit about your background as a sysadmin. + +@3wc: Yeah, absolutely! I've been working on systems administration +mostly on Linux and a little bit of Windows for 10-15 years. Depends +exactly where you start. Within Autonomic cooperative, I'm one of the +founding members and I've been somewhat involved in the infrastructure +over the years. We initiated a project called Coop Cloud, for which +we also host a Gitea instance. I set-up the Co-op Cloud one, and I help +a bit with the maintenance of the Autonomic Co-operative one. + +In terms of general background, I knew Ansible before encountering Gitea. +We currently deploy Gitea using Docker, which I learned through working +on Coopcloud. I wasn't familiar with it before we had, it was my +colleague that setup a Docker based Gitea instance for us -- now I +understand the setup well enough to maintain it but at that time Docker +was entirely new to me and didn't have any experience with that. I've run Trac +and (Phabricator](https://en.wikipedia.org/wiki/Phabricator) at a previous job, +and maybe a couplle of other small code forges, and I've used SVN, CVS, +Mercurial as a developer. + +@dachary: And your first installation of Gitea was for Autonomic via +Docker, you didn't do it before? + +@3wc: Exactly! And I've also setup GitLab CE for the previous employer +but we didn't end up using it extensively. That is sort of general +overview, is there anything else in particular sysadmin-wise that you'd +be curious about? + +@dachary: yeah, I'm always curious about how you choose to work on +something. So you are in a cooperative where you have more leeway than a +usual company but how does that happen? + +@3wc: Maybe it would be useful to tell you about the process to arrive +on Gitea. + +@dachary: Sure! + +@3wc: When we started Autonomic, we had no capital to speak of and our +client income was growing slowly. The amount of time and money that we +had for own infrastructure was minimal. So we started out on GitLab free +account and I think we signed up long enough ago that they gave us quite +a generous free plan and this was before the set of serious ethical +issues with GitLab came out. And I think before git.coop was widespread +through the [Cotech network](https://coops.tech) so we chose GitLab without +knowing either of those things + +@dachary: I see + +@3wc: while using those hosted services, we tried to have a plan to + migrate into self-hosted infrastructure. Partly +because of values and partly because we are trying to host +infrastructure for others and we need to build those skills. Our +impression of Gitlab was that was very heavy to self-host and complicated. And +also none of us particularly liked the interface, we found it slow and +confusing and we are still currently using GitLab for collaboration with +one of our clients that was already using it, and we are +still getting confused by the interface. + +So that was one of the huge +advantage -- there were really two things that led us to Gitea, or led +my colleague towards Gitea a couple of years ago when we first set it +up. It is incredibly simple to run, and it is really fast. + +The interface is a lot more straightforward, it is more familiar to GitHub +or the other kind of ticket tracking systems we've used. The +navigation has fewer layers and we were all impressed with how +quick the interface was, even though it is not doing a +lot of AJAX requests, many of the navigation operations involved full +page loads, it's still much faster than GitLab in our experience. And +I've also noticed the same with the Git operations commonly when I'm +working with my colleague, it will take like 5-10 seconds for them to be +able to push something up to GitLab whereas Gitea is pretty +instantaneous. And then, Gitea has sort of been our test bed because it +was the first application that my colleague setup using our -- first +attempt at the tech stack for Coopcloud which was using -- have you +ever heard of Dokku? + +@dachary: yes + +@3wc: We used it as sort of open source Heroku. So we had a version of Gitea that was +initially hosted through that. My colleague was experimenting with using +that as a framework for hosting a variety of apps. We switched, I think, +at the beginning of last year to Docker Swarm -- maybe slightly longer +than that, maybe mid-2020, we switched to Docker Swarm and we built +Coopcloud on that. I think Gitea was the first +application we packaged using that format, and is the longest running +sort of test bed. And that's still how both https://git.autonomic.zone +and https://git.coopcloud.tech are running using that Docker +Swarm configuration. + +@dachary: Yeah, that is very helpful to know how you came to what is +currently in place. You mentioned that initially you had an +installation, you were looking for forges for your clients and you +started using GitLab but it was too heavy. Now you are using Gitea, can +you give a rough number of clients for which you have -- you manage +Gitea instances? + +@3wc: That's an interesting thing about our current setup: we only have +two Gitea instances and one of them is for Coopcloud and as soon as we +were successful in starting Coopcloud federation then that will, I +guess, no longer be Autonomic hosted. All of our client projects are on +our main Gitea instance. We don't yet have a client who needed their own +private forge instance. We've used private organisations and private +repositories within our existing instances for hosting clients. + +@dachary: And so you say it's interesting because you seem to say that +initially you thought you would create instance for your clients but it +turned out not to be the case? Or did I misunderstand? + +@3wc: I don't know if we knew what the major need would be for git hosting. +No client has approached us with the combination of needs that added up to +they have to have a private Gitea instance. Also, you wanted to ask about our +experience maintaining Gitea, and I think there was a little bit more fragility than we +were expecting, that I think scared us off running multiple instances +but I don't know if I certain that we definitely had intentions to run +many instances right from the get go. I think it has all been a bit +experimental. + +@dachary: I get the impression because this you've done a wonderful job +having generic Ansible stack that can be self-hosted, etc. So it +suggests that you were into self-host for the benefit of clients that's +probably from there(!verify: 11:10) + +@3wc: It's a really interesting observation. I think I'm +stumbling over my words a bit because I just haven't really thought +about it before but -- actually there have been two software development +projects where it was a software co-op that approached us and was +considering getting a Gitea instance. + +I think it was a combination of us having not found it that easy to host +for a couple of reasons that I will get into and the existence git.coop +and other kind of decent hosts. + +I think the main thing that has been on our minds as what we've wanted +to do with Gitea hosting is offering individually. So it -- either as a +benefit to members, or as a kind of service to compete with people +hosting their personal stuff on GitHub or GitLab. So, like a software as +a service model. + +@dachary: So you say that one of the reasons maybe why you don't have +self-hosted Gitea instances managed is because you found that it's +fragile or you had experiences that scared you off? + +@3wc: Yeah, exactly! And the first thing I'll tell you that the thing +has improved over time. The main issue, that was the longest-running +and caused the most time taken up is that until few months ago, Gitea +had an issue where if it started and a single sign-on(SSO) provider wasn't +available then it would disable that single sign-on provider so you'd +have to login as the Gitea administrator and re-enable it but that's now +been fixed. The remaining issue that that's plaguing us is because we +haven't spent the time to dig into is we are not setting +session cookies correctly. So every time there's been a restart, +everyone gets logged out. + +And then there's a conceptual issue which +raises the bar for hosting and I'm not even sure what this solution +would be, I don't know if this is something that you've considered for +your effort or if it's why you are learning towards VPS' over containers +for Gitea hosting is that our current model is to pack apps quite densely +onto single VPS, which works for large majority of +apps that as soon as an app needs network service that doesn't do server name + indication, for instance SSH as faras I know doesn't do server name indication, +so you can't have a reverse proxy listening on the SSH port and directing +requests to the appropriate container without having so kind of +additional layer on the SSH protocol. I think you do that with +things like [stunnel](https://www.stunnel.org/) but then requires +configuration on the client-side. + +So there's a limitation -- what we currently do is we have a different port for connecting to the server to +manage it than we do for Gitea's own SSH server. The problem is that we +would then need to do some manual reconfiguration for every additional +Gitea server that we added to that to be able to use port 2223, 2224, +whatever for each of the additional servers. And that means that if a +client approaches us and says, "we'd like our own Gitea server" then +we'd either need to say, "Okay, but you can't do SSH" or that we need to +invest the time and set-up and maintenance annoyance in +adding another port to our reverse proxy configuration which is +frustratingly difficult. And I remember some other flakiness in towards +the beginning of using Gitea around upgrades, I think the switch from +the normal to the rootless deployment was a bit of a fiasco which +happened sometime last year but yes, that's sort of what I meant in +terms of fragility in the kind of friction. + +@dachary: ah, I see and more recently there has been issues with the +1.16 with the 2FA transition to webauth which maybe you don't use so you +don't notice but people noticed. So yea, I clearly see what you mean. +Okay, this is very interesting. I will move to our series of more +precise questions regarding various topics and starting with contract +and pricing. So I understand that your client do not -- you do not bill +your client for a specific slice of the Gitea hosting but it's part of +the service you provide them? Did you put price tag on that, more or +less, even internally? + +@3wc: The short answer is no, the slightly longer answer is that we have +one organisation that is making a monthly donation in order to use git.coopcloud.tech, +I think that maybe have three repositories, two users or something like that and they are giving us, I +believe, £10 a month for the hosting. + +@dachary: Okay, well that's an interesting -- that's a number +(!verify: 17:18). + +@3wc: All of the others, as you say, it's just rolled into the overheads +of offering sort of software. + +@dachary: So you mentioned that there are -- how many clients your use +your Gitea instance? A rough number (!verify: 17:38) + +@3wc: There's four clients accessing it directly and we have code for a +further three on our Gitea instance. + +@dachary: Okay. So interesting that you have clients who are aware and +using it and your other clients where you use it for their benefit but +they don't use it, correct? + +@3wc: Exactly, yep. + +@dachary: Okay. So for the clients who, so this is moving to the +technical and support part of the questions, so the clients that +actually use Gitea: how do they report issues to you, how does that +happen? + +@3wc: Thinking. It's been a combination of us entering issues based on +them emailing us a list of requests or a file with a lists of requests +and on one hand and on the other hand, clients entering issues directly +and us triaging. It varies with different clients. + +@dachary: Okay, and you say entering issues directly: are they using +Gitea to enter issues or other... + +@3wc: Yeah + +@dachary: Okay. And they are using your -- their own Gitea repository to +file issues or you have a dedicated one for client issues? + +@3wc: The client -- there's one client on each of our Gitea instances +that's directly entering issues and they both have, in those respective +cases, one repository they use. They file issues +directly in those repositories. So we don't have a separate issues-only +repository, it's in with the code. + +@dachary: Okay yeah, fine. And as you say it's only one person who enter +the issues themselves. So you don't have a generalized system for all +the needs of all the customers. It's a case by case basis, correct? + +@3wc: Exactly! I would say that we have a little bit more defined +practice around issue entry and triage systems for the client where we still use +GitLab, because those clients were using it already. + +@dachary: Okay, I see. So you mention it that I will repeat it, just in +case I didn't listen correctly, when they have feature requests or an +improvement request, they also file an issue. It's the same for problems +and feature requests + +@3wc: Yes, exactly. And then we label in Gitea to differentiate +between those. + +@dachary: Okay, fine. Correct. Moving on to the really deep, technical +things: could you describe step-by-step, any level of detail that you +prefer, how the installation process of a Gitea instance goes? How would +you guide someone who wants to do it using your stack? + +@3wc: Yeah. So there's a bit of a decision tree that occurs into it, +which the main question is whether they need a separate VPS and if they +do, we would setup a new VPS using Ansible playbooks which would +configure the file, and install Docker, setup user that kind of thing(a +SSH user account, that is Unix users) and then once that's done or if we +decide that there is a suitable server running already, for instance for +a couple of clients we already run a VPS for them with some other Co-op +Cloud Docker swarm apps, so in those situations it might make sense to +setup Gitea on that. + +So the server is set-up. For the app, the screencast I sent in Matrix runs +through the process. To very briefly summarize that: now, you'd run +a series of commands using abra, which is the Co-op cloud command-line +interface. Within Autonomic, the standard is to run that on one's own +workstation and to interact with a shared repository or environment +variable files representing different application. So at Autonomic we +have a single Git repository on our Gitea, representing all of our +clients, our cloud app configurations. If you run commands to create a +new Gitea instance, it will create a new file in that directory which we +check into Git, push out to the central repository. And then we'd run +some further commands which would connect via SSH to the designated +virtual server and deploy the Gitea instance by downloading Gitea, +MariaDB, whatever the Docker image is necessary, starting them, setting +configuration based on environment variables that we defined along the +way and then going into the Gitea user interface to configure a single +sign-on and... + +@dachary: may I briefly interrupt to (!verify: 24:03) what I +understand + +@3wc: Yes, please! + +@dachary: So from the workstation, you run abra and then abra will -- +you designate a VPS where Gitea has been installed and abra carries +environment variables and will download Gitea so it's all abra doing +until you arrive at the web interface of Gitea, is it correct or is +there some minute steps automated via + +@3wc: Exactly! + +@dachary: And then + +@3wc: By the time you arrive at the user interface, it will have +provisioned a user using a secret that is displayed, it's generated and +displayed during the abra setup process. So the default is a secure +setup. It's not like Wordpress where when you finish an install anyone +can go to the URL and complete the setup. + +@dachary: Okay. So then you go to the web interface and... + +@3wc: Yeah, I mean this is one thing that I did want to touch on. I will +say Gitea seems the best, the easiest of any of the tools that we used +to configure single sign-on providers. The fact that you can do it in +the user interface and that it is very clear how you can add different +authentication providers. We tend to use Keycloak for single sign-on, so +there would be a couple of steps on the Keycloak side to generate an +open ID client for Gitea, to get the client ID and the secret for the +Gitea side of the configuration and I'm trying to think if there was +anything else -- oh, and the other thing that we commonly do is setup a +Drone instance to provide CI/CD for Gitea and that we would also through +abra and the similar process of committing the generated application +environment variable files to Git. + +@dachary: Okay. And when was the last time you deployed a combination of +Gitea and Drone, for instace? + +@3wc: I think it was when we setup https://git.coopcloud.tech/, which +was July last year, something like that. + +@dachary: Okay. And side question: did you consider Woodpecker? + +@3wc: I don't know what that is + +@dachary: It's a fork of Drone. Okay so you did not. + +@3wc: Before the license change, is that correct? + +@dachary: Honestly, I don't know the history. I just know that Drone is +not very actively maintained and Woodpecker is very lively at the moment + +@3wc: Interesting! + +@dachary: It is mostly compatible but not quite. Anyhow, that was a side +question. Okay. And + +@3wc: I think I heard about it briefly through Cyberia, because I think +they were considering it. They are currently using concourse and they +heard we were using Drone and they suggested a fork, we haven't looked +into it. + +@dachary: Okay. And I suppose in the process of this installation there +are also backups? + +@3wc: Yes, we tend to enable server-level backups on the VPS as one +strategy and then -- I don't think it is supported for Gitea yet but we +are rolling out application-level backups for Coopcloud apps, which will +give you like a archive of the files directory and an archive of the SQL +dump in a convinient format rather than have to restoring an entire +operating system image. + +@dachary: So when you say a VPS-level backup, it's like you have the +copy of the QCOW image or something like that? + +@3wc: Yes, exactly! Where we are hosting it on our own infrastructure. +Where we are using a commercial VPS provider, then we use their +snapshot features. + +@dachary: Okay, correct. (!verify: 29:58). What about +monitoring? It's also done by abra's side effect of the installation? + +@3wc: It depends, there's a level of service monitoring that is provided +by Docker Swarm in terms of automatically restarting failed processes. +We are currently rolling out a stack based on Grafana and Prometheus, +haven't been involved since setting it up so I'm not so confident on the +details, for monitoring Coopcloud apps. On top of that, we also use +UptimeRobot for ping and uptime monitoring. + +@dachary: And you have any kind of intrusion detection setup? Something +that would check and give (!verify: 29:53) files, that kind +of things? + +@3wc: We've used lynis and I've used the rkhunter, +can't remember if we've used that app in Co-op on our servers but we +don't have an automated scanning tool. We are looking into log file +monitoring with Loki which we are hoping might +give us some level of that. + +@dachary: Okay. And last question on the setup: can you tell me a little +bit about how the mail is setup for the Gitea instance? How does the +Gitea instance connect to outgoing mail? + +@3wc: Yeah, I'm just trying to login and check... + +@dachary: It's fine if you don't remember + +@3wc: It's going to be one of two options: either we are using a mailbox +provided by domain host Gandi, or or we are using our self-hosted mailing +stack and I will tell you in one second... + +@dachary: And this is taken care of by the recipe, that abra uses to +deploy Gitea, is part of the automated install + +@3wc: Yes, there's a step during the abra setup where you are editing a +configuration file that is specific to your instance of Gitea, and at +that stage you'd define what the SMTP server and any credentials you +need to access it were. + +Our instance is using a Gandi mailbox, we do also self-host a SMTP +server on the same server. I can't remember why we didn't use that +for Gitea. We would probably choose one or the other depending on +what (!verify: 32:05) + +@dachary: Okay, Great! That concludes all the installation process set +of questions. I have a few questions about maintenance. So you mentioned +having VPS' to run Gitea and you also have a Drone instance, what is the +size of these VPS'? Are there two different VPS'? + +@3wc: Main Autonomic Drone is running on the same server as our Gitea +and that server is a quad-core with 16 GB RAM and... + +@dachary: So it is a physical machine? + +@3wc: No it's just a tremendously expensive VPS. And the Coopcloud Gitea +is also running on the same VPS as the Co-op Cloud build server, +that is a 4GB RAM three core VPS that is on our dedicated hardware. + +@dachary: Excellent! Can you describe how a desaster recovery scenario +would look like? For instance, say the VPS just dies, what happens? + +@3wc: Specifically for Gitea? + +@dachary: Yeah, yeah of course! I mean it's the same as every other +because it's VPS based, so... + +@3wc: Yeah, so you say the main server that Gitea is running on dies, on +Autonomic Gitea, we'd provision a new VPS using our Ansible playbooks +and...oh, actually no! I guess, because we -- That's a virtual server, +we just use the features in our VPS host, Hetzner, to provision a new +VPS from a snapshot backup before it died. The Coopcloud instance that's +-- as I said, we are running a dedicated server to host the VPS server, +as you said, we have a virtual image disk backup so we'd use our backup +tool, I think it's restic on that, to restore that +disk image to an earlier version. + +@dachary: Okay, good. And you are -- you rehearse these kind of disaster +recovery? + +@3wc: Not deliberately though, it happened more than once, we deleted the +wrong server by mistake so... + +@dachary: Okay, I see. Yeah. But this is good because I'm always +concerned about disaster recovery not happening often. To your clients, +what kind of SLA do you tell them, for the clients that use your Gitea +instance? What kind of SLA do you promise them? + +@3wc: We have SLAs for some clients, we do GItea stuff with some clients +but as far as I know, there is no overlap between those two groups. Just because +the two clients that are most concerned about uptime don't have Gitea +with us because, in one case they were already on BitBucket for several +years and had no desire to move and on the other case, they don't host +much code and I think they are currently hosting on Gitlab. + +I think we might have a default SLA on our +website, I'll try to find that now but I can also tell you what our +actual actual uptime is + +@dachary: It wouldn't be a separate SLA, it might be a part of other +services' SLAs + +@3wc: Right, yeah. I don't find it on our website. I'll just quickly +tell you what our Gitea was because that might be useful + +@dachary: Okay, sure! + +@3wc: Doesn't seem like uptimerobot is monotiring this actual Gitea +instance, can that really be that case? I get actual monitoring of other +things on the same service. That's an interesting gap. Okay, I can't +tell you the uptime, sorry. + +@dachary: Okay, yeah. If you are using it multiple times a day, you are +the monitoring service! In terms of personalization -- so you have these +two instances, I saw that you added a logo, you changed a few bits -- +could you describe what kind of personalization you did as for as you +remember and how you did it? + +@3wc: I -- yeah, I don't think we did much at all! There's some really +urgent personalization things that we'd like to do, that login +experience with single sign-on is really confusing and apparently the +way to fix that is through a template change but non of us have found +time for that. So every time you need to sign into our Gitea (which +again is frequently, because it logs us out every time it restarts), you +need to click a button at the bottom of the page which says "OpenID", +there's also another button on the login page that says "OpenID". SO +especially for clients that we've tried to get to use the single +sign-on, it's been incredibly confusing. Right now, I don't think we +necessarily changed the logo on ours -- I think the only customization +is that we've changed the tagline to "Git with solidarity" instead of +"Git with a cup of tea". + +@dachary: Interesting! And, so you mentioned a few times that it's +annoying that sessions expire when you reboot the server, how often do +you reboot the Gitea instance? + +@3wc: We at least weekly, or almost always atleast weekly because we +install apt updates and it's often a Docker update which requires a +restart of DOcker containers or a kernel update which restarting the +whole server. There's also -- oh yeah, and then it's slightly more +frequently than that because then, as well as that, there is also Gitea +updates, which require -- which end up invalidating our sessions. I'm +sure you've even seen the option to fix this in the documentation, just +none of us had the time. Probably I could have fixed it in the time it +took complaining about this on this call. + +@dachary: When you mentioned that, I assumed that the accumulation of +reasons to (!verify: 40:49) more than once a week, so you +(!verify: 40:55) your two Gitea instances that you have Drone +associated with it, are there other services also associated with Gitea, +like Gitea uses the static page generator, developed by Codeberg or +maybe some other third-party service? Or is it just... + +@3wc: I only found out about the static page generator last week, I'd +really love to use it. As far as I know, we are not using any services +that integrate with Gitea. We do use some Gitea webhooks stuff to, yeah, +Matrix notifications for activity in the Coopcloud reposts. I don't know +if that counts, that's one integration that we have. + +@dachary: So that would be a link to the chat in Matrix, from the top of +your head do you remember another one, another webhook that... + +@3wc: That's the only existing one currently -- oh no, I'm sorry there's +a bot in the same channel, which I'm about to send you a link to, which +you can use to create issues but, for the life of me, I can't remember +the syntax for it. But it's a lovely idea, I love the concept. I can +just never remember... + +@dachary: Yeah, same here: I love to learn to use it and then I forget. +That -- not specifically this bot but that kind of bot. That is +something that is very attractive, somehow it does not stick. I don't +know why. Now I have two last questions: one, which is a little bit +bizarre so please indulge me! + +@3wc: I'm intrigued! + +@dachary: Imagine you have a magic wand and so you have to step back +from your practical, very technical self and say you can use this magic +wand just once and you can wish for anything as long as it is related to +Gitea managed hosting so you wave your wand and you have your wish, what +your wish -- what would that wish be? + +@3wc: Just to make sure I understand, anything to do with managed Gitea +hosting? + +@dachary: Yeah, in this, yes + +@3wc: This could be any, any like technical change to Gitea or any +existence of the organization providing it or... + +@dachary: We can be really wild as long as it is connected to this, to +Gitea managed hosting, it can be... + +@3wc: I promise I'm not just saying this because I know you are involved +in it but - federation! Like that's the number one thing making it easier +for someone on one Gitea instance to interact with content on another +Gitea instance to lower the cost of moving off of GitHub or GitLab would +be the killer for me. That, as a user, as an administrator and as a +person trying to maybe make it somewhat financially sustainable for +hosters like that is the number one. + +@dachary: Oh, interesting! Okay, that's -- yeah, obviously I like the +answer. I would have the same wish, I guess.. + +@3wc: And then, number 2 is like really only factor until number one is +solved, but they way that we've worked around number one not existing is +to turn on, open, GitHub and GitLab sign up on our Coopcloud Gitea. So +that we could say that we are not on GitHub or GitLab but you can still +use your account to sign in and it will be almost as easy as you were +interacting with us on GitLab or GitHub. + +@dachary: Yeah, it's one step + +@3wc: Yes, it's one step less but it's still more than would be ideal. +And we had to turn that off because we had an instance with someone +logging in via GitLab and using our Drone instance to mine +cryptocurrency. And I guess, I'm aware that there are some spam +prevention features built into Gitea and it's a lot better than what it +was but what I would love is some very accessible, i.e please not a +CAPTCHA, default option for anti-spam. Because I know there is a CAPTCHA +in the default distribution that, we could just turn that on but when we +did previous usability research with people using screen readers, they +told us in no circumstances should we ever use a CAPTCHA for anything. +So I haven't felt comfortable switching that on so we've just been in +this limbo now where you can almost sign in with those providers but you +get to this frosting error screen in the end because open registration's +disabled. So that would be a number two but obviously if you ditch +federation or once federation happens then the need for that kind of +anti-span would be less because we could just make our instance invite +only again it would still be accessible to a wide community of people +through federation. + +@dachary: Yeah, sure! It has other spam issues but it's different. Okay, +wow thank you! That was the first question. The second one is much +easier, which is, if you had to think about one thing that you find +works really well for how you managed Gitea hosting works, what would +that be? If you had to talk to someone, "Oh we do managed Gitea hosting +and this is something that we really like". + +@3wc: I feel bad saying it but Drone! I think the top of the list would +be Drone, it is really easy to set up, to configure both for the administrator +and the people trying to do CI pipelines. It is incredibly fast, the +output it gives you is tremendous. It's conceptually simple, there's no +separate accounts for it. It just uses the same authentication as your +Gitea. It's blown us all away, I think there are people who wouldn't be +down for doing continuous integration or continuous deployment at all +within Autonomic were it not for how good Drone is. + + +@dachary: Wow, interesting! I'm also very happy about Drone, really, and +Woodpecker works the same ways. So it's just another name for the fork +of the project. So, yeah. I completely agree, I'm also very happy about +it. And, so... + +@3wc: I think that's the worst part of GitLab's interface in particular. +The worst part of GitHub's interface is the continuous deployment stuff as well. +I don't know why they've made it such a hot mess. + +@dachary: Yeah, same here, it's easier. So we are done with the +interview. I thank you very much and before we wrap it, do you have any +question? Last words? + +@3wc: No, I thank you for doing this practice, in trying to build this +offering. Like you said, it's not as common as it should be, I'm really +interested in how you approached. I think it will hopefully improve any +usability I'm doing in future. Yeah, I'm super excited to see what comes +out of this project + +@dachary: Thank you! I will cut the recording and...