Skip Navigation

User banner

mrmanager

@ mrmanager @lemmy.today

Posts
64
Comments
1086
Joined
3 yr. ago

Long time linux user and tinkerer. Currently working as a devops engineer. Very positive to the idea of decentralized internet platforms. :)

  • Arch Linux is a Tamagotchi for IT nerds

    Jump

  • My arch system is so stable it's boring. Year after year, just runs. Doesn't slow down, doesn't install crapware.

    So I dont know. I'm happy with it.

  • Testing server load with login required on old.lemmy.today

    Jump

  • Also, while improving server performance and failure modes under load would help, the “badly-written, aggressive scraper bots that ignore robots.txt and are given loads of network resources clobbering servers” is something that affects many, many different Web servers out there. This isn’t a Lemmy problem, nor even just social media problem. It’s a Web problem.

    100%. Everyone gets this problem and then they put their site behind cloudflare. But what happens when the entire Internet is behind an american company who can decide which sites it likes and which it doesnt?

    Same thing for the Threadiverse. The Threadiverse will let you set up a Threadiverse instance and subscribe to everything, efficiently feed all the posts and comments you want to your instance, the moment they come in. In nice, machine-readable form, rather than in something intended for humans that you have to scrape and post-process. But…it takes more dev effort to set up something specific to the Threadiverse than to just treat it like another website.

    Yeah, the web was intended to be this universal format that would always work on any platform and any operating system. But the downside is clear today. Some estimates say that we already have much more bot activity than human activity on the internet. And we have AI creating content for the last few years...

  • Testing server load with login required on old.lemmy.today

    Jump

  • I totally agree with you actually. Maybe there will be some better solution going forward.

  • Testing server load with login required on old.lemmy.today

    Jump

  • Testing server load with login required on old.lemmy.today

    Jump

  • Absolutely. :)

  • Testing server load with login required on old.lemmy.today

    Jump

  • Hi,

    I could actually reproduce this myself now:

    https://lemmy.today/post/57605044

    This link only works sometimes for me too. I have no idea why so will dig into the server logs. :)

  • Testing server load with login required on old.lemmy.today

    Jump

  • I could but there are so many requests, like 30-50 per second, and even if I could cache the login page, I would still spend the bandwidth to send it back to each of those bots. So because of that, and because its not even the main lemmy.today site, I figured its OK to just return 403.

    Server load has decreased to almost nothing on old.lemmy.today. During the years we have been online, more and more bots are scraping and now its just ridicolous. :)

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Testing server load with login required on old.lemmy.today

  • Bots overloading old.lemmy.today

    Jump

  • No, I checked and we do require email. I wasnt sure what was set there but yep, its required. But just put any email address there if you dont want to use a real one. You just wont get an email with approval or denyal after account creation, so you have to come back later and try to log in at some point.

  • Bots overloading old.lemmy.today

    Jump

  • It's fine to not give an email address. It's just for the email to let you know that you were approved or denied. So if you put something random there and check if you can login later, it's all good.

    I think it's even optional to give it... You can try registering without. :)

  • old.lemmy.today is now behind Anubis for bot protection

    Jump

  • Lemmy federation has been known to be buggy in certain releases before, and I really dont know what to do about it except try to keep up to date about bugs. I guess its a complex system underneath. From what ive read, its a queue of federation work that is supposed to take place, and stay in the queue until its successful. But I wish there was a good way to see the status of it.

    I have been looking in db tables and log files before to figure out the current status and it works, but its not so convenient.

  • old.lemmy.today is now behind Anubis for bot protection

    Jump

  • Yeah the API should be exempt. Will look into this a bit more in the coming days but wanted to see how well Anubis works on old.lemmy.today first for a while. We probably end up using it on lemmy.today as well. Just didnt want to make a big bang change everywhere on day 1. :)

  • Bots overloading old.lemmy.today

    Jump

  • Yeah, this is just a start. :)

  • old.lemmy.today is now behind Anubis for bot protection

    Jump

  • Hope so!

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    old.lemmy.today is now behind Anubis for bot protection

  • Bots overloading old.lemmy.today

    Jump

  • I added Anubis for https://old.lemmy.today/ and will see if it helps the situation. Bots are hammering it pretty hard but they are not getting any lemmy content at least. Thank you for your advice. :)

  • Bots overloading old.lemmy.today

    Jump

  • Now it should work well again. I put it behind anubis and we will see if it improves the situation with the bots.

  • Bots overloading old.lemmy.today

    Jump

  • Ok now its working and seemingly blocking bots, but we will see over time how well this works. Like you said, im sure its possible to bypass but maybe it improves the situation somewhat still.

  • Bots overloading old.lemmy.today

    Jump

  • Yeah, I had to take it down overnight and currently experimenting with anubis for it... will be a bit unstable today.. :)

  • Bots overloading old.lemmy.today

    Jump

  • It was down overnight and today im working on trying to get anubus to work with it, so it will be unstable... :)

  • Bots overloading old.lemmy.today

    Jump

  • Its not really working yet, it keeps popping up the "seeing if you are a bot" message all the time.

    But yeah, working on it. If it can prevent dumb bots, its a win for sure.

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Bots overloading old.lemmy.today

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Disk maintenance

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Another theme update

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Blocking broken instances

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Hetzner disk maintenance tomorrow 9.40 to 10.45 CET (03.40 to 04.45 AM in Orlando)

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Default theme change

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Feeling cyberpunk?

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Lemmy.today upgraded to 8 cores, 16 GB

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Another ddos attack... again. :)

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Opinions wanted!

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    Another ddos attack...

  • Announcements @lemmy.today
    mrmanager @lemmy.today

    Ongoing bot attack

  • Linux @lemmy.ml
    mrmanager @lemmy.today

    Cosmic Beta on September 25th!

    system76.com /cosmic
  • Announcements @lemmy.today
    mrmanager @lemmy.today

    Turned off alternate user interfaces temporarily

  • Announcements @lemmy.today
    mrmanager @lemmy.today

    Cert issues has been fixed now

  • Announcements @lemmy.today
    mrmanager @lemmy.today

    Lemmy software updated (0.19.6 -> 0.19.11)

  • LemmyToday @lemmy.today
    mrmanager @lemmy.today

    A community of like-minded people

  • Announcements @lemmy.today
    mrmanager @lemmy.today

    We had unexpected downtime tonight