Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GCP access token expires for services used less frequently #1323

Open
soumyadipDe opened this issue Jun 28, 2024 · 4 comments
Open

GCP access token expires for services used less frequently #1323

soumyadipDe opened this issue Jun 28, 2024 · 4 comments
Labels
bug Something isn't working GCP GCP related issues and PRs

Comments

@soumyadipDe
Copy link
Contributor

Title: GCP access token expires for services used less frequently

Description:

Describe your idea. We are running cartography for our big GCP environment. The issue we faced is that GCP token is expiring for services that are used less frequently. The issue was as below

  • Cartography uses GCP python library to fetch credentials
  • Credetials are service specific i.e. different for compute/storage/dns/container/servicecatalog
  • If a specific credential is not used for 1hr, the token gets revoked. If it is in use, it gets refreshed.
  • Now resources like GKE or GCS buckets are few and far between projects.
  • So interval between two GKE/GCS API call might be more than 1hr for which credentials gets stale and errors out.

To Reproduce:

As said, same service call between projects with gap of more than an hour will cause this.
Logs:
If applicable, copy and paste your console log with the failing stack trace.

ERROR:cartography.sync:Unhandled exception during sync stage 'gcp'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 111, in run
    stage_func(neo4j_session, config)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 252, in start_gcp_ingestion
    _sync_multiple_projects(neo4j_session, resources, projects, config.update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 210, in _sync_multiple_projects
    _sync_single_project(neo4j_session, resources, project_id, gcp_update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 181, in _sync_single_project
    storage.sync_gcp_buckets(neo4j_session, resources.storage, project_id, gcp_update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 247, in sync_gcp_buckets
    storage_res = get_gcp_buckets(storage, project_id)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 32, in get_gcp_buckets
    res = req.execute()
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 932, in execute
    headers=self.headers,
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 222, in _retry_request
    raise exception
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1725, in request
    conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1444, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1396, in _conn_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2570)
Traceback (most recent call last):
  File "/usr/local/bin/cartography", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 724, in main
    sys.exit(CLI(prog='cartography').main(argv))
  File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 705, in main
    return cartography.sync.run_with_config(self.sync, config)
  File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 189, in run_with_config
    return sync.run(neo4j_driver, config)
  File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 111, in run
    stage_func(neo4j_session, config)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 252, in start_gcp_ingestion
    _sync_multiple_projects(neo4j_session, resources, projects, config.update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 210, in _sync_multiple_projects
    _sync_single_project(neo4j_session, resources, project_id, gcp_update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 181, in _sync_single_project
    storage.sync_gcp_buckets(neo4j_session, resources.storage, project_id, gcp_update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 247, in sync_gcp_buckets
    storage_res = get_gcp_buckets(storage, project_id)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 32, in get_gcp_buckets
    res = req.execute()
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 932, in execute
    headers=self.headers,
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 222, in _retry_request
    raise exception
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1725, in request
    conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1444, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1396, in _conn_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2570)

Please complete the following information::

  • Cartography release version - 0.91
  • Python version: 3.10
  • OS: debian
@soumyadipDe soumyadipDe mentioned this issue Jun 28, 2024
Merged
@soumyadipDe
Copy link
Contributor Author

Added suggested PR which worked for my case

@chandanchowdhury
Copy link
Collaborator

Hey Saumyadip, thank you for the bug report.

Are you sure the issue is with token expiration?
Asking cause I would expect HTTP 401 (Unauthorized) when token has expired instead of SSLError with KRB5_S_TKT_NYV.

@soumyadipDe
Copy link
Contributor Author

Yes I did get 401 in another stackthread but did not save if unfortunately. Only when I got 401 I understood the issue and could fix it.

@chandanchowdhury
Copy link
Collaborator

Note: Might be related to #245

@chandanchowdhury chandanchowdhury added GCP GCP related issues and PRs bug Something isn't working labels Jun 30, 2024
@soumyadipDe soumyadipDe mentioned this issue Jul 1, 2024
Draft
chandanchowdhury added a commit that referenced this issue Oct 11, 2024
@soumyadipDe @chandanchowdhury
Adding token generation before each service API (#1324)
05916cd 
Fix for #1323

Co-authored-by: i_virus <[email protected]>
danbrauer pushed a commit to etsy/cartography that referenced this issue Oct 31, 2024
@soumyadipDe @chandanchowdhury @danbrauer
Adding token generation before each service API (cartography-cncf#1324)
0d0cbed 
Fix for cartography-cncf#1323

Co-authored-by: i_virus <[email protected]>
Signed-off-by: Daniel Brauer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working GCP GCP related issues and PRs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chandanchowdhury @soumyadipDe