Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Twitter token multiplexing with database #44

Open
wants to merge 252 commits into
base: master
Choose a base branch
from
Open

Twitter token multiplexing with database #44

wants to merge 252 commits into from

Conversation

notconfusing
Copy link

I refactor the twitter_connect.py module to store tokens in the database, not in memory, for threadsafety.

mmou and others added 30 commits April 16, 2017 01:08
@mmou
Merge branch 'mmou-twitter' of github.com:c4fcm/CivilServant into mmo…
db3d121 
…u-twitter
@mmou
add twitter models, connect, controller
c4c814a
@mmou
make string types bigger, redo alembic, other small changes
6e5612b
@mmou
add TwitterUserState.PROTECTED
50158a3
@mmou
partial work and notes, committing so i can switch branches
b2ce785
@mmou
basic work to refactor connect classes. probably doesn't pass all tests
3ed1b1b
@mmou
split twitter, lumen code into 4 main controller methods. rough work …
2dfa0ea 
…- passes all tests, but tests aren't comprehensive.
@mmou
make query_and_archive_tweets work. nothing else confirmed to work.
2d251e3
@mmou
5 controller methods seem to work.
132a844
updated alembic scripts to keep dev, production, and test synchronized
e948782
@mmou
make twitter lumen code work. add schedule_twitter_jobs.py. tests don…
40b3fc3 
…'t work.
fix bugs. redo alembic again...
5b21661
@mmou
fix small bugs in lumen twitter code
85a0837
edit alembic versions. add some log statements.
3c1441d
@mmou
fix fetch_twitter_tweets bugs. add backfill option. fix random typos
51dd05f
add tweets backfill option. fix other small things.
42d12d7
Merge branch 'mmou-twitter' of https://github.com/mitmedialab/CivilSe…
a90a8c1 
…rvant into mmou-twitter

Conflicts:
	app/controllers/twitter_controller.py
@mmou
twitter observational analysis basic profiling code
3829fa3
@mmou
initial work on TwitterObservationalAnalysisController
54129a0
@mmou
data migration code to remove duplicate twitter user records.
741d6bf
@mmou
fixes to twitter_controller, twitter_observational_analysis_controller.
fd8e4fc
merge diffs in twitter_observational_analysis_controller
7e8234c
@mmou
fix twitter/lumen bugs. update all tests. all tests pass. anonymize f…
bf59e44 
…ixture data.
@mmou
make twitter queries less expensive.
729d0c7
@mmou
start on email_db_report.generate_twitter_report. untested.
03d1d19
@mmou
add tests for twitter_observational_analysis_controller. could be ext…
3e40beb 
…ended.
@mmou
merge with master. fix alembic. all tests pass.
b7e6949
@natematias
added twitter token multiplexing support
761696d
@mmou
remove merge text from email_db_report
1593ddd
@mmou
Merge branch 'mmou-twitter' of github.com:c4fcm/CivilServant into mmo…
0a271cc 
…u-twitter
notconfusing and others added 30 commits June 12, 2020 11:41
@notconfusing
abandon tu-et join strategy, use internal guratantee instead
e0563e1
@notconfusing
update email db report
ee1b111
@notconfusing
use <br> to make readable
2aba958
@notconfusing
report fixes and schedule fixes
2ed1416
@notconfusing
handle error where twitter response may have changed
c6c0752
@notconfusing
add failed queue count to db report
cbe38d0
@notconfusing
dynamic control group sizes
fe2d301
@notconfusing
fix misspelling
be1d36d
@notconfusing
need a mu
94d126d
@notconfusing
some extra add retryables
f4ae664
@notconfusing
run the filler more often for les time
209c271
@notconfusing
run the filler more often for les time
cc8c7e8
@notconfusing
timedelta needs seconds
1cd4c63
@notconfusing
add retryable to fill query
a5513a9
@notconfusing
add fills
edca2c0
@notconfusing
add in backfills
10de0d7
@notconfusing
Merge remote-tracking branch 'origin/mmou-twitter' into mmou-twitter
1e1c070
@notconfusing
log exceptions with log.exceptions
99a86c1
@notconfusing
log info on shorter running things
a3796e3
@notconfusing
start adding things for the new observation study
e502b13
@notconfusing
update cmd
3212e2c
@notconfusing
Merge branch 'mmou-twitter' of https://github.com/mitmedialab/CivilSe…
ee23070 
…rvant into mmou-twitter
@notconfusing
fix header being written on retry
99d626f
@notconfusing
rand min and max from commandline
a2dc5ae
@notconfusing
check done users just within range
7df5f71
@notconfusing
add back in mysteriously missing snapshots
29594f2
@notconfusing
fetch tweets more often
6ae55a9
@notconfusing
Merge branch 'mmou-twitter' of https://github.com/mitmedialab/CivilSe…
16cabee 
…rvant into mmou-twitter
@notconfusing
just archive some processed users, not everyone ever.
8236729
@notconfusing
more retryability
409fd6a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@epenn epenn Awaiting requested review from epenn

@natematias natematias Awaiting requested review from natematias

Assignees

@notconfusing notconfusing

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@notconfusing @mmou @natematias @pushshift @jonathanzong