Link: https://theredarchive.xyz/
Preview: https://i.imgur.com/BXXQOke.png & https://i.imgur.com/niDdoEW.png
As a web developer that discovered TRP 1 year ago and is very grateful for the subreddit, I've always wanted to contribute here, but I never knew how, until now. After TRP has been quarantined, I feared it would get banned one day. So I decided to figure out a way to scrape the entire subreddit and have it viewed on a simple website.
I saw TRP's current backup of the subreddit and I wasn't happy with its design and hard-to-use website (and its lack of posts). So I decided to spend 8 hours trying to figure out how to scrape the entire subreddit and then code a website to view the posts as simple as possible.
Features:
- 4392 posts from TheRedPill & askTRP & RedPillParenting & RedPillWomen & ThankTRP
- Comments (+ replies) included
- TRP's subreddit theme
- Search through all the posts instantly
- Lightweight and simple to use website & no ads
- DDoS protected and secured
NOTE: The purpose of the website is to only have an archive of TRP posts. This is not a place for people to post or comment. If you want to do that if TRP ever gets banned, visit the official website at https://www.forums.red/i/
To Mods: If this post is not allowed, please let me know and I'll remove it instantly. My intention is to only contribute to TRP and spread the idea to men around the world, I am in no way trying to harm this subreddit.
Edit: Thank you for the amazing feedback everyone. I just finished scrapping RedPillWomen & RedPillParenting & ThankTRP and added them both to the website. Up to 1.5k posts added.
redpillschool Admin 5y ago
Thanks. So did we. https://www.forums.red/i/theredpill
We will always welcome more backup points, so thank you for helping out.
Our backup includes 3,176 Posts between TRP and AskTRP. With over 519,619 individual comments.
Counting RPW and ThankTRP, our total archive is 4,573 posts.
Instant search can be found here
In case of emergency we will release the entire database as torrent. FYI.
Forums opening soon with new mobile-friendly design.
drakehfh 5y ago
It would be good to make the site open source with all the database. That way people would contribute.
throne_deserter 5y ago
TRP changed my life. Quarantine had me scared but I am glad that you guys are doing all this for those who might need help.
RainySeasonInPH 5y ago
Not sure if you guys are aware, the entire reddit post history is backed up on google biquery. I've used it to dredge old redpill posts. I'm pretty sure there's also a torrent.
shownothirstyo 5y ago
Yours looks great, its the go to back up. His however is quite a bit faster and very sleek, a collaboration would be nice.
redpillschool Admin 5y ago
While we are working on proper caching at the moment, I'm legitimately curious what makes you say that.
https://www.forums.red/i/theredpill generates in 0.2027 seconds. A page with 345 comments generated in 0.1854 seconds.
A side-by-side load test shows my pages serve a bulk of the page 1 full second quicker (aside from post-loaded JS and images, which are specifically set to load slowly)
http://a.trp.red/files/timing.jpg
I'm hoping to streamline this so please let me know.
WarViper1337 5y ago
With the way things are going having extra back ups is probably a good thing.
Mgtow_Maester 5y ago
Any chance you could do that for mgtow as well? Not to mention some other manosphere subs.
redpillschool Admin 5y ago
https://www.reddit.com/r/TheRedPill/comments/9mr07o/our_open_invitation_to_mods_join_our_life_raft/
dream-hunter 5y ago
Not sure, not a big fan of that sub. If more TRP people want that then I can make it happen.
[deleted]
standardmissile 5y ago
Brilliant. Apart from the practical benefit this is a great example of DOING rather than WHINGING in response to the quarantine.
Some readers here really are internalising TRP and it's fantastic to see. Well done OP.
duehvdke 5y ago
You're doing god's work my guy, keep it up!
MortalSisyphus 5y ago
This is amazing. It would be great to have this done on my subreddit, which is obviously at high risk of being banned. Since you already developed the code, would it be particularly difficult to set this up for another sub? Please let me know.
redpillschool Admin 5y ago
https://www.reddit.com/r/TheRedPill/comments/9mr07o/our_open_invitation_to_mods_join_our_life_raft/
lemonized 5y ago
Thank you! This is just so compact and simply amazing. Contains every type of sorting and a live search. Amazing work. Never hurts to have more backups just in case I guess.
SalesOverEverything 5y ago
Just wanted to thank you as well. This makes a real impact on men everywhere, usually for the better.
​
Thank you.
YesToControversy 5y ago
Have this virtual beer, mate. Don't drink it and drive though!
????
JerryAwesome 5y ago
Awesome, thanks!
ImmunosuppressedTau 5y ago
Thanks man!
​
El_Ejcovero 5y ago
You can never have too many backups, especially with the purging of "controversial" content and people online as of late.
ubisoft-vs-ea 5y ago
You are amazing, if I wasn’t poor I’d give you gold
[deleted]
wereworm5 5y ago
You are the hero we all wanted !
_Neon_Shadow_ 5y ago
Amazing! I'm not in love with trp.red but this is an excellent substitute. Thank you.
[deleted]
[deleted]
DulceDeLecheMardel 5y ago
Can you make it downloadable so we can mirror it?
[deleted]
huhub 5y ago
I've been working on one of TRP backups, tried to post here, but the post did not make it.
AskTRP announce is here: asktrp/comments/9jxdtf/trp_sub_offline_backup/
​
I have completed 2016 and 2017, 2013-2015 are in progress, and I'm going to periodically update 2018. The entire archive contain 68,000+ posts + uncountable comments, no scripts and other unnecessary information (which reduced the size by half at least).
ReeZoX 5y ago
Really like it!
​
Only thing I would like is being able to sort/search the results based on the categories (flairs) and then sort that with the most upvoted one's :)
dream-hunter 5y ago
Doing that is possible. Click on the Category column on the table two times then you'll see the most upvoted posts on that flair (or type the entire category's name in the search function). You can even see the most upvoted posts of each subreddit; just click the Subreddit column.
Thank you for the feedback!
ReeZoX 5y ago
The search function works, but I can't switch the category on the column, there's either "Science" or none/unspecified displaying for me there ;)
​
dream-hunter 5y ago
Refresh page > click Column tab two times then you'll see the posts from each category in order of most upvoted.
fuckboiwithfeelings 5y ago
The community that keeps on giving, way to go!
dream-hunter 5y ago
Looking forward to contributing more. If you have more ideas about TRP for a web developer, let me know!
redpillschool Admin 5y ago
I can invite you to our private dev forum (open soon). Instead of duplicating efforts we have a lot of tasks that need doing.
RedPillHanSolo 5y ago
I've sent you numerous e-mails, but haven't heard back. Is it because you didn't get them or you only invite trusted members and whatnot?
redpillschool Admin 5y ago
I haven't forgotton about you. I simply haven't gotten the private forum running yet. Lots of work going on in the background.
[deleted]
[deleted]
Magnus_ORily 5y ago
I feel part of an ancient civilisation who's history has been preserved
BlueFreedom420 5y ago
Thank you for this. Keep the memory before the big blackout (when the state and the elites to finally take the Earth)
johnpayne10 5y ago
Dude, excellent job. The website interface is really smooth, it loads very quickly. I want to give two suggestions though.
First: Maybe you shouldn't put asktrp posts in there? Most of the posts in asktrp are not worth anyone's time. It is a good thing to provide advice to someone asking for help. But I don't think it is necessary to add asktrp questions to the archive site. Add more TRP posts if you can. Because the posts and the comments sections on TRP are invaluable. If you already have added the top posts from all time, add the ones from this current year. Add the hot posts. The new posts.
Second: For the posts that you have uploaded only a part of the comments section is avaIlable. If possible, try to add the entire comments section for all posts. Because their is a lot of gold in the comments.
That being said, you have done an excellent job. It will help a lot of people out there. So congratulations and thank you.
izzyinjurious 5y ago
What language did you use to scrape it? It's awesome btw good work.
dream-hunter 5y ago
PHP, my favorite language. :) Thank you!
McDrMuffinMan 5y ago
Wow.... You're either a masochist or something else.
God speed Man!
TheTriviaMan 5y ago
I write php myself and even though you've already done the scraping I would suggest for any future developers to use a python library known as "beautiful soup" https://pypi.org/project/beautifulsoup4/ it's made specifically for web scraping
SpiderAlpha33 5y ago
Yeah BeautifilSoup is fast and reliable, and for dynamically generated websites I use Selenium with headless Firefox.
the-dan-man 5y ago
Out of curiousity, why is PHP your favourite language, and where did you learn it?
dream-hunter 5y ago
I don't remember, but long ago I wanted to start coding my own screenshot software, and my friend told me I could do that with my back-then favorite language & PHP, so I started learning PHP.
I learnt PHP by coding, and whenever I wanted to code something, I searched on Google about it and read the code. PHP is really simple.
ThePantsThief 5y ago
Why scrape it when they have an API? Fellow developer here. Curious in case the scraping solution is better somehow
redpillschool Admin 5y ago
The https://forums.red/i/TheRedPill archive was done via API
needz 5y ago
Sometimes if you already have your favorite tool and it works, there's no need to learn another tool (or in this case, API).
SilkTouchm 5y ago
Scraping is always harder than just using the api.
needz 5y ago
Never say never or always.
So if I've been using BeautifulSoup since it came out, have an entire framework and dev environment dedicated to scraping websites and have little to no experience with APIs, which is gonna be harder?
SilkTouchm 5y ago
Scraping, by far. It doesn't matter how much experience you have, you still have to do the legwork. An API is just using a few pre-packaged methods, you don't need experience on it.
Brushyourteethm8 5y ago
Solid work, thanks! Will you be doing the same for AskTRP, MRP and AskMRP? Some solid posts and advice in each
dream-hunter 5y ago
I'll run my scrapper on all the Red Pill subreddits from the sidebar, I kinda forgot about those.
They'll be on the site within 48hr.
Brushyourteethm8 5y ago
Good effort, thanks
lonefireinwater 5y ago
Thank you! This is amazing.
robodylan123 5y ago
Did something similar here: http://trpbackup.com I have every single post backed up but only the top 1000 are browsable at the moment. You’ll be able to search all of them soon though.
yesbuthereswhy 5y ago
Great website but is there an option to search by date?
redpillschool Admin 5y ago
Our search: https://www.forums.red/i/TheRedPill/?search=reddit&submitsearch=Search
has a date setting.
dream-hunter 5y ago
Just added. Check the date column and click on it to sort. https://i.imgur.com/kCQf3h9.png
[deleted]
hardlifeman 5y ago
Thanks for doing this.
When I click on these "load more comments" in old posts, it opens up a seperate "about:blank" page.
redpillschool Admin 5y ago
Until he fixes it, forums.red does have exhaustive comment chains.
dream-hunter 5y ago
Load more comments, unfortunately, doesn't work, as I wasn't able to scrape those.
redpillschool Admin 5y ago
This might help:
https://www.reddit.com/dev/api#GET_api_morechildren
dream-hunter 5y ago
Thank you, looks good. Unfortunately, I didn't scrape the posts using Reddit's API. I might be able to find another way to do it, though.
redpillschool Admin 5y ago
Gotcha, good luck.
bousezz 5y ago
sub posts can be formatted as a document. which document format do you prefer ?
html, pdf, epub, and plain text.
i am a software developer.
[deleted]
Asktrpthrowaway420 5y ago
Really awesome, clean and easy to navigate
[deleted]
[deleted]
[deleted]
A_Bandini 5y ago
Beautiful work dude. Thank you for this. The sacred archives cannot be lost